Memory Usage after recent updates has increased significantly and is now unstable

Guys, this is driving me up the wall!

Once again, after having my system running stable for more than a month, it appears that something in a recent update has suddenly caused my system memory to spiral out of control again.

Memory usage went from a steady 21-23% to now almost 40% and fluctuating all over the place!

I do not have this problem with any other servers other than this one running Virtualmin (i also have a Centos7 system running Centos Web Panel).

How can i troubleshoot this problem?

Status: 
Active

Comments

Howdy -- thanks for contacting us!

It looks like you have a good bit of RAM free, and fluctuations aren't abnormal at all. There's a lot of potential causes, though something like a bot hitting a website could definitely cause that.

Also note that we haven't issued any updates recently, any updates would have been from Debian itself.

We'd be happy to review your running processes though, and see if anything appears out of the ordinary.

What is the output of this command:

ps auxwf

# ps auxwf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0      0     0 ?        S    Sep26   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Sep26   0:11  \_ [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [kworker/0:0H]
root         7  0.0  0.0      0     0 ?        S    Sep26   1:09  \_ [rcu_sched]
root         8  0.0  0.0      0     0 ?        S    Sep26   0:00  \_ [rcu_bh]
root         9  0.0  0.0      0     0 ?        S    Sep26   0:02  \_ [migration/0]
root        10  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [lru-add-drain]
root        11  0.0  0.0      0     0 ?        S    Sep26   0:02  \_ [watchdog/0]
root        12  0.0  0.0      0     0 ?        S    Sep26   0:00  \_ [cpuhp/0]
root        13  0.0  0.0      0     0 ?        S    Sep26   0:00  \_ [cpuhp/1]
root        14  0.0  0.0      0     0 ?        S    Sep26   0:02  \_ [watchdog/1]
root        15  0.0  0.0      0     0 ?        S    Sep26   0:02  \_ [migration/1]
root        16  0.0  0.0      0     0 ?        S    Sep26   0:12  \_ [ksoftirqd/1]
root        18  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [kworker/1:0H]
root        19  0.0  0.0      0     0 ?        S    Sep26   0:00  \_ [kdevtmpfs]
root        20  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [netns]
root        21  0.0  0.0      0     0 ?        S    Sep26   0:01  \_ [khungtaskd]
root        22  0.0  0.0      0     0 ?        S    Sep26   0:00  \_ [oom_reaper]
root        23  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [writeback]
root        24  0.0  0.0      0     0 ?        S    Sep26   0:00  \_ [kcompactd0]
root        26  0.0  0.0      0     0 ?        SN   Sep26   0:00  \_ [ksmd]
root        27  0.0  0.0      0     0 ?        SN   Sep26   0:10  \_ [khugepaged]
root        28  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [crypto]
root        29  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [kintegrityd]
root        30  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [bioset]
root        31  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [kblockd]
root        32  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [devfreq_wq]
root        33  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [watchdogd]
root        35  0.0  0.0      0     0 ?        S    Sep26   0:00  \_ [kswapd0]
root        36  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [vmstat]
root        48  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [kthrotld]
root        49  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [ipv6_addrconf]
root        84  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [bioset]
root        85  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [ata_sff]
root        86  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [bioset]
root        87  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [bioset]
root        88  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [bioset]
root        90  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [bioset]
root        92  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [bioset]
root       117  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [bioset]
root       118  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [bioset]
root       119  0.0  0.0      0     0 ?        S    Sep26   0:00  \_ [scsi_eh_0]
root       120  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [scsi_tmf_0]
root       121  0.0  0.0      0     0 ?        S    Sep26   0:00  \_ [scsi_eh_1]
root       122  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [scsi_tmf_1]
root       124  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [bioset]
root       125  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [bioset]
root       130  0.0  0.0      0     0 ?        S<   Sep26   0:13  \_ [kworker/1:1H]
root       148  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [kworker/u5:0]
root       158  0.0  0.0      0     0 ?        S    Sep26   0:50  \_ [jbd2/vda1-8]
root       159  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [ext4-rsv-conver]
root       197  0.0  0.0      0     0 ?        S    Sep26   0:00  \_ [kauditd]
root       207  0.0  0.0      0     0 ?        S<   Sep26   0:04  \_ [kworker/0:1H]
root       283  0.0  0.0      0     0 ?        S    Sep26   0:00  \_ [hwrng]
root       299  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [ttm_swap]
root       364  0.0  0.0      0     0 ?        S<   Sep26   0:00  \_ [edac-poller]
root      6665  0.0  0.0      0     0 ?        S    06:44   0:00  \_ [kworker/0:1]
root      9045  0.0  0.0      0     0 ?        S    07:29   0:00  \_ [kworker/u4:1]
root     13862  0.0  0.0      0     0 ?        S    08:09   0:00  \_ [kworker/1:2]
root     14518  0.0  0.0      0     0 ?        S    08:14   0:00  \_ [kworker/0:5]
root     17795  0.0  0.0      0     0 ?        S    08:29   0:00  \_ [kworker/1:0]
root     17880  0.0  0.0      0     0 ?        S    08:29   0:00  \_ [kworker/0:0]
root     17899  0.0  0.0      0     0 ?        S    08:29   0:00  \_ [kworker/0:2]
root     21597  0.0  0.0      0     0 ?        S    08:39   0:00  \_ [kworker/1:4]
root     21599  0.0  0.0      0     0 ?        S    08:39   0:00  \_ [kworker/1:5]
root     23805  0.0  0.0      0     0 ?        S    08:44   0:00  \_ [kworker/u4:2]
root     25069  0.0  0.0      0     0 ?        S    08:49   0:00  \_ [kworker/0:3]
root     25083  0.0  0.0      0     0 ?        S    08:49   0:00  \_ [kworker/0:4]
root     25109  0.0  0.0      0     0 ?        S    08:49   0:00  \_ [kworker/1:1]
root     25638  0.0  0.0      0     0 ?        S    08:54   0:00  \_ [kworker/u4:0]
root         1  0.0  0.1 204672  6752 ?        Ss   Sep26   5:15 /sbin/init
root       186  0.0  0.1  59352  7128 ?        Ss   Sep26  16:30 /lib/systemd/systemd-journald
root       231  0.0  0.0  45484  3116 ?        Ss   Sep26   0:01 /lib/systemd/systemd-udevd
root       319  0.0  0.1  46496  4760 ?        Ss   Sep26   3:45 /lib/systemd/systemd-logind
message+   323  0.0  0.0  45116  3676 ?        Ss   Sep26   3:45 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root       341  0.0  0.8 200720 34816 ?        Ssl  Sep26   0:20 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid
root       343  0.0  0.0  35924  3248 ?        Ss   Sep26   0:29 /usr/sbin/irqbalance --foreground
root       344  0.0  0.0  27988  2468 ?        Ss   Sep26   0:05 /usr/sbin/cron -f
root       345  0.0  0.0 250112  4040 ?        Ssl  Sep26   2:23 /usr/sbin/rsyslogd -n
root       352  0.0  0.0   4172    76 ?        S<   Sep26   0:31 /usr/sbin/atopacctd
root       371  0.0  0.0   8428   104 ?        Ss   Sep26   0:00 /sbin/rdnssd -u rdnssd -H /etc/rdnssd/merge-hook
rdnssd     373  0.0  0.0  16868  1512 ?        S    Sep26   0:00  \_ /sbin/rdnssd -u rdnssd -H /etc/rdnssd/merge-hook
root       420  0.0  0.1  79696  5452 ?        Ss   Sep26   1:07 /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root       421  0.0  0.1  79696  5452 ?        S    Sep26   1:07  \_ /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root       422  0.0  0.1  79696  5452 ?        S    Sep26   1:07  \_ /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root       423  0.0  0.1  79696  5452 ?        S    Sep26   1:07  \_ /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root       424  0.0  0.1  79696  5452 ?        S    Sep26   1:07  \_ /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
postgrey   425  0.0  0.5  74520 20988 ?        Ss   Sep26   0:00 postgrey --pidfile=/var/run/postgrey.pid --daemonize --inet=10023
bind       546  0.0  0.9 289508 36828 ?        Ssl  Sep26   2:24 /usr/sbin/named -f -u bind
root       569  0.0  0.0  14524  1716 tty1     Ss+  Sep26   0:00 /sbin/agetty --noclear tty1 linux
root       585  0.0  0.1  69948  5672 ?        Ss   Sep26   0:00 /usr/sbin/sshd -D
root       612  0.0  0.0  18124  2540 ?        Ss   Sep26   0:01 /usr/sbin/dovecot
dovecot    628  0.0  0.0   9492   996 ?        S    Sep26   0:00  \_ dovecot/anvil
root       629  0.0  0.0   9624  2324 ?        S    Sep26   0:00  \_ dovecot/log
ntp        631  0.0  0.1 102416  4052 ?        Ssl  Sep26   1:01 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 106:111
root       647  0.0  0.0  20472  3052 ?        Ss   Sep26   0:00 /sbin/dhclient -4 -v -pf /run/dhclient.ens3.pid -lf /var/lib/dhcp/dhclient.ens3.leases -I -df /var/lib/dhcp/dhclient6.ens3.leases ens3
root       862  0.2  0.9 1213880 40000 ?       Sl   Sep26  58:47 /usr/bin/python3 /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b
mysql      923  0.1  5.4 662684 219920 ?       Ssl  Sep26  26:39 /usr/sbin/mysqld
root       945  0.0  2.1 166432 87460 ?        Ss   Sep26   2:39 /usr/bin/perl -T -w /usr/sbin/spamd -d --pidfile=/var/run/spamd.pid --create-prefs --max-children 5 --helper-home-dir
root      1152  0.0  2.0 166432 82180 ?        S    Sep26   0:03  \_ spamd child
root      1153  0.0  2.0 166432 82180 ?        S    Sep26   0:02  \_ spamd child
root      1080  0.0  1.8  98800 73640 ?        Ss   Sep26   0:00 /usr/share/webmin/virtual-server/lookup-domain-daemon.pl
root      1090  0.0  0.1 287168  7328 ?        Ssl  Sep26   0:49 /usr/lib/policykit-1/polkitd --no-debug
root      1138  0.0  0.1  81636  4512 ?        Ss   Sep26   0:31 /usr/lib/postfix/sbin/master -w
postfix   1142  0.0  0.1  83872  6956 ?        S    Sep26   0:02  \_ qmgr -l -t unix -u
postfix   1196  0.0  0.2  88176  8924 ?        S    Sep26   0:04  \_ tlsmgr -l -t unix -u -c
postfix   1658  0.0  0.1  83704  6848 ?        S    Sep26   0:22  \_ anvil -l -t unix -u -c
postfix   8658  0.0  0.1  83704  6920 ?        S    07:19   0:00  \_ pickup -l -t unix -u -c
postfix  18885  0.0  0.2 107292 10856 ?        S    08:33   0:00  \_ smtpd -n smtp -t inet -u -c -o stress=yes -s 2 -o smtpd_sasl_auth_enable=yes
postfix  19029  0.0  0.2 107292 10480 ?        S    08:33   0:00  \_ smtpd -n smtp -t inet -u -c -o stress=yes -s 2 -o smtpd_sasl_auth_enable=yes
postfix  24728  0.0  0.2 107292 11000 ?        S    08:48   0:00  \_ smtpd -n submission -t inet -u -c -o stress=yes -s 2 -o smtpd_sasl_auth_enable=yes
postfix  25253  0.0  0.2 107292 10628 ?        S    08:49   0:00  \_ smtpd -n submission -t inet -u -c -o stress=yes -s 2 -o smtpd_sasl_auth_enable=yes
postfix  25392  0.0  0.2 107428 10584 ?        S    08:52   0:00  \_ smtpd -n smtps -t inet -u -c -o stress= -s 2 -o smtpd_sasl_auth_enable=yes -o smtpd_tls_wrappermode=yes
postfix  25393  0.0  0.1  83704  6900 ?        S    08:52   0:00  \_ proxymap -t unix -u
root      1154  0.0  0.5  79828 22768 ?        Ss   Sep26   0:13 /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root      1157  0.0  0.5  85540 22476 ?        Ss   Sep26   0:45 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root     25766 37.0  2.3 159172 95848 ?        S    08:54   0:00  \_ /usr/share/webmin/shell/index.cgi
root     25767  0.0  0.0   4276   796 ?        S    08:54   0:00      \_ sh -c (ps auxwf) 2>&1
root     25768  0.0  0.0  36768  2952 ?        R    08:54   0:00          \_ ps auxwf
root     17265  0.0  0.2 315288 11548 ?        Ssl  Sep26   0:47 /usr/lib/packagekit/packagekitd
root     25396  0.0  0.0  89952   464 ?        Ss   Sep26   0:00 gpg-agent --homedir /root/.gnupg --use-standard-socket --daemon
root      9522  0.0  0.9 393676 40160 ?        Ss   Oct09   0:21 php-fpm: master process (/etc/php/7.0/fpm/php-fpm.conf)
ajecrea+  9524  0.0  6.5 647488 267096 ?       S    Oct09   2:49  \_ php-fpm: pool 152349738012302
ajecrea+  9775  0.0  6.4 645656 261756 ?       S    Oct09   2:50  \_ php-fpm: pool 152349738012302
ajecrea+ 10188  0.0  6.7 657492 273828 ?       S    Oct09   2:46  \_ php-fpm: pool 152349738012302
ajecrea+ 17514  0.0  6.6 654660 271100 ?       S    Oct09   2:43  \_ php-fpm: pool 152349738012302
root     15336  0.0  0.2  25344  8416 ?        S<Ls 00:00   0:06 /usr/bin/atop -a -R -w /var/log/atop/atop_20191012 600
root      5023  0.0  0.7 268868 31772 ?        Ss   06:25   0:00 /usr/sbin/apache2 -k start
www-data  5067  0.0  0.6 268592 24532 ?        S    06:25   0:00  \_ /usr/sbin/apache2 -k start
www-data  5068  0.0  0.9 1476876 36684 ?       Sl   06:25   0:07  \_ /usr/sbin/apache2 -k start

Yes i think its a Debian update that has caused this...why is the question?

Is there a way i can download system usage stats before 3 days ago, and then now? (so i can compare then to see what has caused this big increase in RAM usage?)

~]# apache2ctl -M
Loaded Modules:
core_module (static)
so_module (static)
watchdog_module (static)
http_module (static)
log_config_module (static)
logio_module (static)
version_module (static)
unixd_module (static)
access_compat_module (shared)
actions_module (shared)
alias_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_core_module (shared)
authn_file_module (shared)
authz_core_module (shared)
authz_host_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cgi_module (shared)
dav_module (shared)
dav_fs_module (shared)
deflate_module (shared)
dir_module (shared)
env_module (shared)
fcgid_module (shared)
filter_module (shared)
lbmethod_byrequests_module (shared)
mime_module (shared)
mpm_event_module (shared)
negotiation_module (shared)
proxy_module (shared)
proxy_balancer_module (shared)
proxy_connect_module (shared)
proxy_fcgi_module (shared)
proxy_http_module (shared)
reqtimeout_module (shared)
rewrite_module (shared)
security2_module (shared)
setenvif_module (shared)
slotmem_shm_module (shared)
socache_shmcb_module (shared)
ssl_module (shared)
status_module (shared)
suexec_module (shared)
unique_id_module (shared)

Well, I'm really not seeing anything out of the ordinary there, that looks perfectly normal.

Using more memory also doesn't necessarily indicate a problem, it could just mean a service sees that more RAM is available and is doing more caching. Or it could mean something like a PHP module is using more. Those sorts of things would be tricky to track down without knowing how much RAM was being used by each service previously.

While you can see some general usage stats in the Virtualmin "System Statistics" area, it doesn't store anything that specific.

Unfortunately, unless you explicitly stored that information, there really wouldn't be anything to compare to.

However, I'm really not seeing signs of a problem there though. Sorry, but I'm not sure there's much we could do to assist.

hmmm, this is interesting...i went to ajecreative virtual server, then changed php-fpm to fcgi, then back to php-fpm again (this restarted php-fpm).

Now look at my stats...

https://drive.google.com/file/d/1xj1t-dlpApHnd0byFuiqVepvXv_H3WSA/view?u...

It will be interesting to see if this stabilizes, or if it starts to gradually climb. I will update you guys again once i have some firm idea.

You can use an open source monitor tool like Netdata, that is extremely light. With it you can decide for how much time of statistics you want to preserve, for example, for the last 15 days.

As Andrey said, with this information is a lot easier to track scenarios like yours.

https://www.netdata.cloud/