"Error establishing a database connection" - how to troubleshoot this?

11 posts / 0 new
Last post
#1 Mon, 04/21/2014 - 11:46
christophera

"Error establishing a database connection" - how to troubleshoot this?

Ugh - how do I figure this one out?

On Virtualmin GPL, I have several wordpress sites that, since last night, have been responding intermittently with "Error establishing a database connection" when accessed.

One moment they load fine, the next I get the error. Happens in both front and back end.

However, I can access the database with phpMyAdmin repeatedly with no trouble at all. Everything pops right up, no errors, no tables are reported as broken....

Suggestions?

Virtualmin 4.06 CentOS Linux 6.5 mysql 5.1.73 WP 3.9 (been running 3.9 for about a week, no problems until last night)

Mon, 04/21/2014 - 12:44
andreychek

Howdy,

Hmm, do you see a more detailed error in $HOME/logs/error_log? Or is that the full error that you receive?

-Eric

Mon, 04/21/2014 - 13:11 (Reply to #2)
christophera

Hi Eric,

Thank you for getting back to me.

I'm not sure where you mean by $HOME/logs/error_log

Is that the same as etc/httpd/logs/error_log ?

In that, I'm getting:

PHP Warning: PHP Startup: Unable to load dynamic library 'ssh2.so > /etc/php.d/ssh2.ini' - ssh2.so > /etc/php.d/ssh2.ini: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Directive 'register_globals' is deprecated in PHP 5.3 and greater in Unknown on line 0 PHP Warning: Directive 'register_globals' is deprecated in PHP 5.3 and greater in Unknown on line 0 [Mon Apr 21 05:36:56 2014] [notice] caught SIGTERM, shutting down [Mon Apr 21 05:37:22 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Mon Apr 21 05:37:22 2014] [notice] Digest: generating secret for digest authentication ... [Mon Apr 21 05:37:22 2014] [notice] Digest: done [Mon Apr 21 05:37:23 2014] [notice] Apache/2.2.15 (Unix) DAV/2 mod_fcgid/2.3.9 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips SVN/1.6.11 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations PHP Warning: Directive 'register_globals' is deprecated in PHP 5.3 and greater in Unknown on line 0 PHP Warning: Directive 'register_globals' is deprecated in PHP 5.3 and greater in Unknown on line 0 Failed loading /usr/local/ioncube/ioncube_loader_lin_5.2.so: /usr/local/ioncube/ioncube_loader_lin_5.2.so: cannot open shared object file: No such file or directory PHP Warning: Directive 'register_globals' is deprecated in PHP 5.3 and greater in Unknown on line 0 Failed loading /usr/local/ioncube/ioncube_loader_lin_5.2.so: /usr/local/ioncube/ioncube_loader_lin_5.2.so: cannot open shared object file: No such file or directory PHP Warning: Directive 'register_globals' is deprecated in PHP 5.3 and greater in Unknown on line 0 PHP Warning: Module 'mcrypt' already loaded in Unknown on line 0 PHP Warning: Module 'mcrypt' already loaded in Unknown on line 0 [Mon Apr 21 16:10:02 2014] [notice] caught SIGTERM, shutting down [Mon Apr 21 16:10:28 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Mon Apr 21 16:10:28 2014] [notice] Digest: generating secret for digest authentication ... [Mon Apr 21 16:10:28 2014] [notice] Digest: done [Mon Apr 21 16:10:29 2014] [notice] Apache/2.2.15 (Unix) DAV/2 mod_fcgid/2.3.9 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips SVN/1.6.11 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations PHP Warning: Directive 'register_globals' is deprecated in PHP 5.3 and greater in Unknown on line 0 PHP Warning: Directive 'register_globals' is deprecated in PHP 5.3 and greater in Unknown on line 0 PHP Warning: PHP Startup: Unable to load dynamic library 'ssh2.so > /etc/php.d/ssh2.ini' - ssh2.so > /etc/php.d/ssh2.ini: cannot open shared object file: No such file or directory in Unknown on line 0 Failed loading /usr/local/ioncube/ioncube_loader_lin_5.2.so: /usr/local/ioncube/ioncube_loader_lin_5.2.so: cannot open shared object file: No such file or directory PHP Warning: Directive 'register_globals' is deprecated in PHP 5.3 and greater in Unknown on line 0 Failed loading /usr/local/ioncube/ioncube_loader_lin_5.2.so: /usr/local/ioncube/ioncube_loader_lin_5.2.so: cannot open shared object file: No such file or directory PHP Warning: Directive 'register_globals' is deprecated in PHP 5.3 and greater in Unknown on line 0

I think the ioncube error is from an old install, I'm checking on that. Would the 'caught SIGTERM, shutting down' be related to not connecting to mysql?

Mon, 04/21/2014 - 13:21 (Reply to #3)
christophera

Oh, I didn't get the $HOME/logs/error_log reference because the site I was looking at doesn't have a 'logs/' directory. (should I make one for it, and will it automatically populate it?).

Another site that I have wordpress on does have that directory. Some problem with intermittent database connection errors. There are no related errors that I see though (name/ip changed for privacy):

[Sun Apr 20 21:59:08 2014] [error] [client 108.287.222.85] Directory index forbidden by Options directive: /home/mysite/public_html/wp-content/plugins/wp-symposium/js/ [Mon Apr 21 03:15:33 2014] [warn] [client 108.287.237.20] mod_fcgid: stderr: PHP Warning: include(hashTagbox.php): failed to open stream: No such file or directory in /home/mysite/public_html/wp-content/plugins/wp-front-end-editor-master/class-wp-front-end-editor.php on line 1504, referer: http://www.mysite.com/ [Mon Apr 21 03:15:33 2014] [warn] [client 108.287.237.20] mod_fcgid: stderr: PHP Warning: include(): Failed opening 'hashTagbox.php' for inclusion (include_path='.:/usr/share/pear:/usr/share/php') in /home/mysite/public_html/wp-content/plugins/wp-front-end-editor-master/class-wp-front-end-editor.php on line 1504, referer: http://www.mysite.com/ [Mon Apr 21 03:21:47 2014] [warn] [client 108.287.237.20] mod_fcgid: stderr: PHP Warning: include(hashTagbox.php): failed to open stream: No such file or directory in /home/mysite/public_html/wp-content/plugins/wp-front-end-editor-master/class-wp-front-end-editor.php on line 1504, referer: http://www.mysite.com/ [Mon Apr 21 03:21:47 2014] [warn] [client 108.287.237.20] mod_fcgid: stderr: PHP Warning: include(): Failed opening 'hashTagbox.php' for inclusion (include_path='.:/usr/share/pear:/usr/share/php') in /home/mysite/public_html/wp-content/plugins/wp-front-end-editor-master/class-wp-front-end-editor.php on line 1504, referer: http://www.mysite.com/

Mon, 04/21/2014 - 13:27 (Reply to #4)
christophera

Old (no longer used) ioncube entry in my daughter's site's php.ini found and squashed.

Mon, 04/21/2014 - 22:54 (Reply to #5)
christophera

Was fine for about 8 hours and then started happening again...

no errors in var/log/mysqld.log

load averages a little higher than I would expect, as I'm not getting a traffic spike right now. But not alarmingly high, 1.3 or so on a 4 core server.

Tue, 04/22/2014 - 00:07
andreychek

Howdy,

Yeah, the logs that would be key here are either /home/mysite/logs/error_log, or /var/log/virtualmin/mysite_error_log.

I see a few errors relating to WordPress issues, possibly with WordPress plugins.

However, what specific errors are you seeing in the error logs now whenever it throws a database error?

-Eric

Tue, 04/22/2014 - 01:31 (Reply to #7)
christophera

I'm not seeing any specific errors when I look at the logs, maybe I'm looking at the wrong ones?

My hosting company (DigitalOcean) was nice enough to take a look and I just heard back from them a few minutes ago.

They said:

"I did take a quick look, and I see that there are a large number of idle connections, so I'm sure this is the cause of your problems. I see that these connections are idle for hundreds of seconds, and I'm guessing this is due to your application not closing out the connections when it is done with them, and possibly re-opening new ones."

They recommended lowering the timeout as a temporary solution, then trying to track down the cause.

Does that sound reasonable?

I regularly keep WP updated and update plugins... how would I track down what is leaving open connections (if you agree this is a likely cause...)?

Interesting though, I run the sites through cloudflare and tonight they are reporting a huge (6885) spike in 'threats' on one of my sites.

They don't provide info on what those threats are though, so I don't know what that means. I upped their 'challenge' system and even disabled the site through virtualmin, but my other sites continued to get those errors. I have to think that the spike in threats and the idle connections are related, eh? But shouldn't disabling the site cause this to be fixed pretty quick if this was the root cause?

Sorry to write a book here!

Chris

Tue, 04/22/2014 - 08:58
andreychek

Howdy,

Well, it's possible your site is receiving more hits than it can handle... it could be related to normal traffic, bots trying to break in, or just a general DoS.

You could look at the corresponding access_log to see what kind of connections you're receiving at the time your site isn't working.

It wouldn't hurt to reduce the Apache timeout. I often reduce that from the default of 300 to 30 on my systems.

-Eric

Tue, 04/22/2014 - 23:30 (Reply to #9)
christophera

Done.

Thank you very much for your help on this.

I think the problem ended up being a combination of things - the bad traffic caused problems, and while investigating, I found the idle connections were all on one database (different than the one the bad traffic was reported on in cloudflare), so I shut down that site and all returned to normal.

Thanks again for helping out. It's stressful to deal with these things and your forum and advice always helps,

Chris

Wed, 04/23/2014 - 08:12
andreychek

Oh nuts, I just realized you were referring to idle database connections, rather than idle Apache connections... but I'm glad you got things sorted out!

-Eric

Topic locked