centOS 5.2 Virtualmin GPL (How to know version of Virtualmin? Installed summer 08 or later, it's releatively recent) Real memory 3.38 GB total, 434.63 MB used I'm not much of an admin: not stupid but not a lot of time for this part.
I've had to reboot twice in 24hrs. The web server appears to die slowly choking, rather than keel over. I'm having trouble imagining what changed. I see no obvious traffic spikes or mysql issues.
I see potential issues...
the mailman log is huge and I don't use mailman (how did it get turned on?)...
there is a huge log called btmp I understand I need to add something to get this rotated/managed, but I'm reluctant to delete it before this issue is resolved in case it is the issue (wow, what a lot of attempts to log in).
But mostly I am clueless. I read a lot about configuring Apache/Memory. But I suspect this is a case of "something went wrong", rather than a config or resource availability issue. The config files and hardware are adequate and have not changed.
Anyone have any clues to share?
Here is just about all I know to inspect...
...
Allowed memory size of 44040192 bytes exhausted (tried to allocate 24272896 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 39907328 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 21432320 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 23816192 bytes)
[Mon Feb 01 03:52:15 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
Allowed memory size of 44040192 bytes exhausted (tried to allocate 39907328 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 31250432 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 39907328 bytes)
...
Allowed memory size of 44040192 bytes exhausted (tried to allocate 22226944 bytes)
[Mon Feb 01 04:03:01 2010] [notice] Graceful restart requested, doing restart
Allowed memory size of 44040192 bytes exhausted (tried to allocate 39907328 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 22810624 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 20000768 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 22607872 bytes)
[Mon Feb 01 04:03:20 2010] [notice] Digest: generating secret for digest authentication ...
[Mon Feb 01 04:03:20 2010] [notice] Digest: done
Allowed memory size of 44040192 bytes exhausted (tried to allocate 28248064 bytes)
[Mon Feb 01 04:03:21 2010] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Mon Feb 01 04:03:22 2010] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
Allowed memory size of 44040192 bytes exhausted (tried to allocate 22628352 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 20000768 bytes)
...
Allowed memory size of 44040192 bytes exhausted (tried to allocate 39907328 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 23021568 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 39907328 bytes)
[Mon Feb 01 04:30:03 2010] [notice] Graceful restart requested, doing restart
Allowed memory size of 44040192 bytes exhausted (tried to allocate 22401024 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 28688384 bytes)
[Mon Feb 01 04:30:46 2010] [notice] Digest: generating secret for digest authentication ...
[Mon Feb 01 04:30:46 2010] [notice] Digest: done
[Mon Feb 01 04:30:49 2010] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Mon Feb 01 04:30:50 2010] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
Allowed memory size of 44040192 bytes exhausted (tried to allocate 25020416 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 20015104 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 19994624 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 28688384 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 20000768 bytes)
...,
Allowed memory size of 44040192 bytes exhausted (tried to allocate 29378560 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 28688384 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 21432320 bytes)
[Mon Feb 01 08:43:02 2010] [notice] Graceful restart requested, doing restart
Allowed memory size of 44040192 bytes exhausted (tried to allocate 39907328 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 25520128 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 28227584 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 20021248 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 22401024 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 31686656 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 31901696 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 22401024 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 27703296 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 39907328 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 19980288 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 22810624 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 28248064 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 20015104 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 22628352 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 30005248 bytes)
[Mon Feb 01 08:43:38 2010] [notice] Digest: generating secret for digest authentication ...
[Mon Feb 01 08:43:38 2010] [notice] Digest: done
[Mon Feb 01 08:43:39 2010] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Mon Feb 01 08:43:39 2010] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
Allowed memory size of 44040192 bytes exhausted (tried to allocate 30025728 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 23529472 bytes)
Allowed memory size of 44040192 bytes exhausted (tried to allocate 39907328 bytes)
# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.3 0.0 10348 76 ? Ss Jan30 11:17 init [3]
root 2 0.0 0.0 0 0 ? S< Jan30 0:00 [migration/0]
root 3 0.0 0.0 0 0 ? SN Jan30 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S< Jan30 0:00 [watchdog/0]
root 5 0.0 0.0 0 0 ? S< Jan30 0:00 [migration/1]
root 6 0.0 0.0 0 0 ? SN Jan30 0:00 [ksoftirqd/1]
root 7 0.0 0.0 0 0 ? S< Jan30 0:00 [watchdog/1]
root 8 0.0 0.0 0 0 ? S< Jan30 0:00 [events/0]
root 9 0.0 0.0 0 0 ? S< Jan30 0:00 [events/1]
root 10 0.0 0.0 0 0 ? S< Jan30 0:00 [khelper]
root 39 0.0 0.0 0 0 ? S< Jan30 1:10 [kthread]
root 44 0.0 0.0 0 0 ? S< Jan30 0:05 [kblockd/0]
root 45 0.0 0.0 0 0 ? S< Jan30 0:00 [kblockd/1]
root 46 0.0 0.0 0 0 ? S< Jan30 0:00 [kacpid]
root 159 0.0 0.0 0 0 ? S< Jan30 0:00 [cqueue/0]
root 160 0.0 0.0 0 0 ? S< Jan30 0:00 [cqueue/1]
root 163 0.0 0.0 0 0 ? S< Jan30 0:00 [khubd]
root 165 0.0 0.0 0 0 ? S< Jan30 0:00 [kseriod]
root 241 6.5 0.0 0 0 ? S< Jan30 200:32 [kswapd0]
root 242 0.0 0.0 0 0 ? S< Jan30 0:00 [aio/0]
root 243 0.0 0.0 0 0 ? S< Jan30 0:00 [aio/1]
root 387 0.0 0.0 0 0 ? S< Jan30 0:00 [kpsmoused]
root 418 0.0 0.0 0 0 ? S< Jan30 0:00 [ata/0]
root 419 0.0 0.0 0 0 ? S< Jan30 0:00 [ata/1]
root 420 0.0 0.0 0 0 ? S< Jan30 0:00 [ata_aux]
root 424 0.0 0.0 0 0 ? S< Jan30 0:00 [scsi_eh_0]
root 425 0.0 0.0 0 0 ? S< Jan30 0:00 [scsi_eh_1]
root 426 0.4 0.0 0 0 ? S< Jan30 14:16 [kjournald]
root 453 0.0 0.0 0 0 ? S< Jan30 0:00 [kauditd]
root 487 0.0 0.0 12604 4 ? S<s Jan30 0:00 /sbin/udevd -d
mysql 717 8.1 0.9 629624 33028 ? Sl Jan31 171:14 /usr/libexec/my
mailman 1384 0.3 0.0 151652 1324 ? S Jan31 6:41 /usr/bin/python
mailman 1412 0.2 0.0 151544 1324 ? S Jan31 5:50 /usr/bin/python
root 1428 0.0 0.0 0 0 ? S< Jan30 0:00 [kmpathd/0]
root 1429 0.0 0.0 0 0 ? S< Jan30 0:00 [kmpathd/1]
mailman 1436 0.2 0.0 151604 1316 ? S Jan31 4:32 /usr/bin/python
root 1459 0.0 0.0 0 0 ? S< Jan30 0:13 [kjournald]
root 1461 0.1 0.0 0 0 ? S< Jan30 4:25 [kjournald]
root 1463 0.0 0.0 0 0 ? S< Jan30 0:00 [kjournald]
mailman 1464 0.2 0.0 151560 996 ? S Jan31 4:18 /usr/bin/python
mailman 1534 0.1 0.0 151648 1356 ? S Jan31 2:27 /usr/bin/python
mailman 1678 0.0 0.0 151556 1876 ? S Jan31 1:16 /usr/bin/python
mailman 1682 0.0 0.0 151560 1876 ? S Jan31 1:22 /usr/bin/python
mailman 1692 0.0 0.0 151576 1916 ? S Jan31 0:16 /usr/bin/python
root 1947 0.3 0.0 5908 372 ? Ss Jan30 11:49 syslogd -m 0
root 1950 0.0 0.0 3800 136 ? Ss Jan30 1:55 klogd -x
named 1987 0.7 0.9 239752 34376 ? Ssl Jan30 21:48 /usr/sbin/named
dbus 2011 0.0 0.0 21268 4 ? Ss Jan30 0:00 dbus-daemon --s
root 2055 0.0 0.0 60596 320 ? Ss Jan30 0:14 /usr/sbin/sshd
root 2091 0.0 0.0 65932 4 ? S Jan30 0:00 /bin/sh /usr/bi
postgres 2218 0.2 0.0 120596 336 ? S Jan30 9:04 /usr/bin/postma
postgres 2220 0.0 0.0 109772 156 ? S Jan30 0:27 postgres: logge
postgres 2222 0.1 0.0 120596 200 ? S Jan30 4:46 postgres: write
postgres 2223 0.3 0.0 110776 64 ? S Jan30 11:33 postgres: stats
postgres 2224 0.0 0.0 109908 348 ? S Jan30 0:02 postgres: stats
nobody 2317 0.5 0.0 49312 580 ? Ss Jan30 15:56 proftpd: (accep
root 2374 0.4 0.0 75692 396 ? Ss Jan30 13:30 crond
root 2389 0.0 0.0 46716 4 ? Ss Jan30 0:00 /usr/sbin/sasla
root 2392 0.0 0.0 46716 4 ? S Jan30 0:00 /usr/sbin/sasla
root 2393 0.0 0.0 46716 4 ? S Jan30 0:00 /usr/sbin/sasla
root 2394 0.0 0.0 46716 4 ? S Jan30 0:00 /usr/sbin/sasla
root 2395 0.0 0.0 46716 4 ? S Jan30 0:00 /usr/sbin/sasla
68 2408 0.4 0.0 30612 560 ? Ss Jan30 14:00 hald
root 2409 0.0 0.0 21668 4 ? S Jan30 0:00 hald-runner
68 2416 0.0 0.0 12308 4 ? S Jan30 0:00 hald-addon-acpi
mailman 2517 0.0 0.0 149488 656 ? Ss Jan30 0:00 /usr/bin/python
root 2536 0.0 0.0 123896 8 ? Ss Jan30 0:56 /usr/libexec/we
root 2550 0.4 0.0 74472 1068 ? Ss Jan30 14:19 /usr/bin/perl /
root 2555 0.4 0.0 71920 2208 ? Ss Jan30 13:53 /usr/bin/perl /
root 2558 0.0 0.0 3788 4 tty1 Ss+ Jan30 0:00 /sbin/mingetty
root 2559 0.0 0.0 3788 4 tty2 Ss+ Jan30 0:00 /sbin/mingetty
root 2560 0.0 0.0 3792 4 tty3 Ss+ Jan30 0:00 /sbin/mingetty
root 2561 0.0 0.0 3788 4 tty4 Ss+ Jan30 0:00 /sbin/mingetty
root 2562 0.0 0.0 3792 4 tty5 Ss+ Jan30 0:00 /sbin/mingetty
root 2563 0.0 0.0 3792 4 tty6 Ss+ Jan30 0:00 /sbin/mingetty
root 14146 0.0 0.0 88884 3360 ? Ss 11:44 0:00 sshd: root@pts/
root 14151 0.0 0.0 66068 1620 pts/1 Ss 11:44 0:00 -bash
root 14760 0.0 0.0 102900 1184 ? S 12:00 0:00 crond
root 14767 0.0 0.0 8696 1040 ? Ss 12:00 0:00 /bin/bash /usr/
root 14774 0.0 0.0 3784 432 ? S 12:00 0:00 sleep 8868
root 14945 0.0 0.0 87992 3336 ? Ss 12:05 0:00 sshd: root@pts/
root 14947 0.0 0.0 66064 1588 pts/0 Ss 12:05 0:00 -bash
root 14969 0.1 0.0 12608 1048 pts/0 S+ 12:05 0:00 top
root 14990 0.0 0.0 65596 968 pts/1 R+ 12:07 0:00 ps aux
root 16429 0.0 0.0 0 0 ? S 04:53 0:00 [pdflush]
root 16851 0.0 0.0 0 0 ? S 09:39 0:00 [pdflush]
[root@dex221 httpd]#
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 221920 1587412 31172 1531984 86 151 655 279 35 66 21 24 49 6 0
[root@dex221 httpd]#
Howdy,
Yeah, Mailman is enabled in a default Virtualmin install. If you and your users don't need it, I might suggest disabling it. In Virtualmin, you can go into System Settings -> Features and Plugins, and disable Mailman. Further, you'll want to stop the Mailman service from running -- which you can do in Webmin -> System -> Bootup and Shutdown.
At far as the btmp file goes, while it might be worth looking into that at some point, large files wouldn't typically cause any problems up until the point that your disk fills up. You can check that by running "df -h".
As far as the above processes go -- could you by chance run "ps auxw", and attach the output as a file to this thread? Some of the text in your 'ps' output above is getting cut off, hopefully this will solve that.
The "Allowed memory size of 44040192 bytes exhausted" message is typically seen in PHP applications that are trying to do something, but running out of memory before they complete.
What I'd recommend doing there is editing your php.ini file (in /etc/), and setting the "memory_limit" option to a higher value.
It looks like your application is requiring more than 44MB of RAM -- so you might consider making the limit 64MB, or maybe even 128MB.
Lastly -- 2008 is a long time, in Internet years :-)
A lot has gone on in development of Virtualmin since then, I'd suggest upgrading to a recent version. A lot of resource usage improvements have gone in there... and also a few security fixes.
You can tell what version you have when you log into Virtualmin, by looking at the System Information screen that shows up, and checking out the value next to "Virtualmin version".
-Eric