Impossible to login to PHPMyadmin - and other Apache sites

40 posts / 0 new
Last post
#1 Mon, 08/08/2011 - 03:29
jvieille

Impossible to login to PHPMyadmin - and other Apache sites

I suddenly lost the ability to connect or re-connect to Apache sites. The sites still work correctly, either in connected or not connected modes.

Particularly, I can no longer connect to PHPMyAdmin - it stays on the login page without any error. There are no error in logs, the connection attempts do not even appear in the logs. It seems that Apache - MySQL do not intereract for login - though once connected, it works mostly fine.

Any clue?

Thank for help, I am totally stuck

Mon, 08/08/2011 - 07:34
Locutus

Uhm, can you clarify this one please? You're talking about "connections" in as I see it several different scenarios. What does "re-connecting to sites" mean?

Also, I think your first two sentences contradict each other... If you can't connect to sites, how are they still "working correctly"?

Please tell us in detail what you tried, what you expected and what happened instead.

Mon, 08/08/2011 - 07:45
jvieille

Connecting - I mean

  • web sites have public and private access - The user that logs in get "connected"

Working correctly:

  • anonymous visitors can browse the web site
  • logged in users can also browse and pubish content

The problem

  • there is no way to log in once "disconnected" or logged out. I cannot login anymore to any web site
  • specifically, it is not possible to login to phpMyAdmin - the main page displays correctly, but nothing happens when submitting username and password - exactly like all web sites
  • no error log nor access information in apache2 logs.

Clearer?

Thank you very much for taking care

Mon, 08/08/2011 - 08:47
Locutus

Okay, I think I get it now. With "connected" you mean "logged in" or "authenticated".

The issue you're seeing might be a browser problem. What happens when you try it with a different web browser? You might also try clearing the browser cache and cookies, and restart it.

Mon, 08/08/2011 - 08:53
jvieille

I tried with IE, FF and Chrome, cleared the caches and so on - same thing. It works correctly with another quite similar server

Mon, 08/08/2011 - 10:15
jvieille

Any place I should look for to solve this critical issue?

Mon, 08/08/2011 - 10:34
andreychek

So, just to clarify -- it's not just you, but all users on your server are experiencing the same problem?

Or are other users able to log into sites on your server?

If you're able to browse to various websites on your server, but not login... and that's a problem all users on your server are experiencing, two thoughts come to mind --

  1. I'm surprised that you're not seeing any messages at all in the error logs, you'd definitely want to make sure you're looking in the logs specific to the domain that's having problems -- that's located in $HOME/logs/error_log.

  2. It's possible that the sites aren't communicating with MySQL. Again, that would normally show up in the error logs, but it can't hurt to restart MySQL by running this from the command line as root: /etc/init.d/mysqld restart

Mon, 08/08/2011 - 11:10
jvieille

I did not ask other users, I tried from different computers from home, same behaviour It looks like mysql is never hit "directly" by my web sites. I even change the database name of one site, it still worked!

There are no error at all, the access log stopped recording anything since yesterday I suspect some cache to bypass all requests from Apache

Mon, 08/08/2011 - 12:58
Locutus

A simple and maybe stupid question: You did try restarting Apache, right? :)

Mon, 08/08/2011 - 13:03
andreychek

Howdy,

If all websites that have a login on your server aren't functioning, is it normal that users wouldn't be complaining to you?

With that large an issue, and it's been broken for at least 8 hours now, it seems unusual that you're the only one who's noticed.

So, I'm suspicious that it may be somehow related to your home setup, or maybe your ISP.

What you're describing certainly could be a cache issue... it's possible that your ISP is using some form of caching proxy that's causing the issues you're seeing.

Do you have someone who can test it using a completely unrelated connection and ISP than you've tried so far?

-Eric

Mon, 08/08/2011 - 13:30
jvieille

After further research, I actually get Apache access info, but not when tryiong to login, and not for certain actions (like asking for a new password)

Only one site could have get soe login attempts today, but not sure. Also, they would normally not need to login again as their session is saved.

I of course restarted Apache, MySQL, and even the server several times. I tried to kill all possible caches with no more success.

Your suggestion about my ISP might be worth exploring, because this is really puzzling

Mon, 08/08/2011 - 13:35
andreychek

Is this something that we could test out?

Is there a URL, username, and password for some sort of test site that you could post here that we could try to see if we get the same results?

-Eric

Mon, 08/08/2011 - 15:43
jvieille

Good idea However, I cannot create a dummy user to let you connect to the web site. But you can at least try these 2 phpMyAdmin installed by VM on 2 different servers

entering any (wrong) login should warn "#xxx Cannot log in to the MySQL server"

1- http://b2mml-services.com/phpmyadmin/

2- http://shareontheweb.com/phpmyadmin1/

On my side:

(1) behave correctly on this server

(2) keep silent on the faulty server.

Also I created a user vmt/vmt that has permission for a useless database "cases" - you should be abale to connect to http://shareontheweb.com/phpmyadmin1 using this login

Thanks a lot

Mon, 08/08/2011 - 17:57
Locutus

The second server fails for me as well, there's no error message or other comment upon login, no matter if I enter nonsense or the correct username. I'm simply taken back to the login screen.

Is the "1" after the second URL correct? Does phpMyAdmin maybe have some local configuration where you need to enter its URL path? Does it match the URL you're using?

Mon, 08/08/2011 - 17:58
Locutus

Does this help maybe? The following post describes an issue with php session variable directory.

http://www.blardone.org/hq/phpmyadmincannotlog-inbutnoerrormessage

Otherwise, I'd suggest thoroughly comparing the Apache and PHP settings on the working and failing servers, including PHP execution mode (mod_php vs. FCGI), the suexec settings and file access rights (ownership/permissions).

Mon, 08/08/2011 - 17:57
andreychek

Yeah, I'm seeing the same results you are... I do receive a "Can't login" message on the first link, and when using the user information you provided on the second url, it returns to a blank screen with no error message.

Did that phpMyAdmin install every work correctly?

And, did you say you're seeing similar problems with other web app logins on your server?

-Eric

Tue, 08/09/2011 - 04:24
jvieille

Locutus:

  • yes, the "1" in http://shareontheweb.com/phpmyadmin1/ is correct

  • phpMyAdmin was installed using VM script installer in both cases. I tried to reinstall several times, it proceeds normally. It used to work, but not anymore

  • I checked the session.save_path issue, tried several configs. What I found is that the directory was empty, but 300K size. I deleted the directory and rcreated it, I also changed the location of this path, but without any effect after restarting Apache, and even rebooting the server.

andreychek:

Every app I need to login through Apache / php fails. I can connect to VM without problem, but cannot do anything with my web sites runing Apache / php, including phpMyAdmin as discussed just above. The sites I am logged in keep as is until I log out - then I have no way to login anymore.

Looking at both servers PHP information, I found that: checking phpinfo on the phpMyAdmin welcome screen, the variable HTTP_COOKIE contents this:

1) failing server

phpMyAdmin unlogged SESS9e15b58ce905bc905f72910fd9ae69ab=6c4f07f435d9c90241f164a3838e9dd0; has_js=1; __utma=73559177.1425389158.1312879513.1312879513.1312879513.1; __utmb=73559177.2.10.1312879513; __utmc=73559177; __utmz=73559177.1312879513.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)

Another web site unlogged SESS9e15b58ce905bc905f72910fd9ae69ab=6c4f07f435d9c90241f164a3838e9dd0; has_js=1; __utma=73559177.1425389158.1312879513.1312879513.1312879513.1; __utmb=73559177.2.10.1312879513; __utmc=73559177; __utmz=73559177.1312879513.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)

(all my sites are unlogged now, as I emtied my browsers' caches :()

2) correct server

phpMyAdmin unlogged phpMyAdmin=hrf0t84o0m545lp7b22g25hv1h28g5hi; pma_lang=en; pma_collation_connection=utf8_general_ci; pma_mcrypt_iv=iyD0j%2BI9cVw%3D

phpmyadmin logged phpMyAdmin=g0l517u3kdg01ogtqo9o322th49is5r3; pma_lang=en; pma_collation_connection=utf8_general_ci; pma_mcrypt_iv=iyD0j%2BI9cVw%3D; pmaUser-1=vDlyo%2BCmVWk%3D; pmaPass-1=U1fA7eleX6o%3D; has_js=1; SESS384300a1104195dd3b58ab2f046bb319=3L6xB2QzMjVoZhwoVrxTosP3FKdO4y55rC_9lRvf0dE; Drupal.toolbar.collapsed=0

Another web site unlogged has_js=1

the same, logged has_js=1; SESS384300a1104195dd3b58ab2f046bb319=3L6xB2QzMjVoZhwoVrxTosP3FKdO4y55rC_9lRvf0dE; Drupal.toolbar.collapsed=0

???

Tue, 08/09/2011 - 05:35
Locutus

Did you compare the configuration for Apache and PHP for the two servers thoroughly? There has to be SOME difference if the systems are otherwise the same.

Tue, 08/09/2011 - 07:02
jvieille

They both were installed by Webmin / Virtualmin, though the failing one was updated with dotdeb for upgrading Mysql to 5.1 - php upgraded to 5.2.17 at this moment.

Apart of phpinfo,

1- http://b2mml-services.com/phpinfo.php

2- http://shareontheweb.com/phpinfo.php

what should I look for?

thanks again

Tue, 08/09/2011 - 09:34
andreychek

Well, what you're describing there all sounds very bizarre.

I'm curious, can you access MySQL from the command line?

For example, if you log into your server as root over SSH, and then run:

mysql -p

Enter your root password when prompted... do you then get to the MySQL shell? Or is there an error of some kind?

Do you know of anything on your system that may have changed prior to all these problems occurring?

-Eric

Tue, 08/09/2011 - 10:01
jvieille

Yes, I can connect to Mysql without any problem The last things I remember before I noticed the change was:

  • a Drupal script running to update the translations that failed by lack of php memory - I then increased from 1G to 2G (I reverted back since then)

  • installing a Drupal forked module that seemed buggy - I removed it later on

  • I also made intensive performance testing (http://www.webpagetest.org/) that put some stress on the system

  • My disk system partition went full during performance tests with XHProf (I changeed the output directory somewer under home/ after

Tue, 08/09/2011 - 12:17
jvieille

I just finnished checking my phpinfo Not much to say though - the mysql client version does not match the mysql server's (this is intentionnal and not an issue according to dotdeb)

  • the cookies behave very differently:

when on the phpadmin login page, - an unlog site generates no cookie on the "good server", - the "bad one" sends cookies

Tue, 08/09/2011 - 16:00
jvieille

I look at the logs, this is what I get when I attempt to logon on both servers

Good

Web site www.b2mml-services.com

80.15.104.74 - - [09/Aug/2011:20:19:22 +0200] "POST /user HTTP/1.1" 302 699 "http://www.b2mml-services.com/user" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"

80.15.104.74 - - [09/Aug/2011:20:19:22 +0200] "GET /user/1 HTTP/1.1" 200 4397 "http://www.b2mml-services.com/user" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"

Phpmyadmin

80.15.104.74 - - [09/Aug/2011:20:30:32 +0200] "GET /phpmyadmin HTTP/1.1" 301 582 "-" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"

80.15.104.74 - - [09/Aug/2011:20:30:40 +0200] "POST /phpmyadmin/index.php HTTP/1.1" 302 750 "http://www.b2mml-services.com/phpmyadmin/" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"

80.15.104.74 - - [09/Aug/2011:20:30:40 +0200] "GET /phpmyadmin/index.php?token=c81fb760be7645690febc6486f7f6d6a HTTP/1.1" 200 1708 "http://www.b2mml-services.com/phpmyadmin/" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"

80.15.104.74 - - [09/Aug/2011:20:30:40 +0200] "GET /phpmyadmin/js/common.js?ts=1312723535 HTTP/1.1" 200 1789 "http://www.b2mml-services.com/phpmyadmin/index.php?token=c81fb760be7645690febc6486f7f6d6a" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"
...

BAD

web site "www.shareontheweb.com"

80.15.104.74 - apc [09/Aug/2011:20:15:26 +0200] "POST /user HTTP/1.1" 200 2550 "http://www.shareontheweb.com/user" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1

phpMyAdmin

80.15.104.74 - - [09/Aug/2011:20:27:25 +0200] "GET /phpmyadmin1 HTTP/1.1" 301 253 "-" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"

80.15.104.74 - apc [09/Aug/2011:20:27:33 +0200] "POST /phpmyadmin1/index.php HTTP/1.1" 200 1397 "http://www.shareontheweb.com/phpmyadmin1/" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"

80.15.104.74 - apc [09/Aug/2011:20:27:33 +0200] "GET /phpmyadmin1/js/messages.php?lang=en&db=&token=bcddc93e16f97da3209b8f21fb53b9a5 HTTP/1.1" 200 4936 "http://www.shareontheweb.com/phpmyadmin1/index.php" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"

80.15.104.74 - apc [09/Aug/2011:20:27:33 +0200] "GET /phpmyadmin1/phpmyadmin.css.php?server=1&token=bcddc93e16f97da3209b8f21fb53b9a5&js_frame=right&nocache=3938189007 HTTP/1.1" 200 41533 "http://www.shareontheweb.com/phpmyadmin1/index.php" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"

(fail to logon)

Note1: I am wondering why "apc" appears just before the timestamp in "BAD".

Note2: apc.php mini-sites works normally - it does not rely on Mysql for login http://www.shareontheweb.com/apc

Tue, 08/09/2011 - 16:18
Locutus

"APC opcode cache"?? Well, did you try disabling that and see if your logins work then?

The text "apc" before the timestamp means that a web user "apc" authenticated with Apache.

Tue, 08/09/2011 - 16:51
jvieille

Yes, I use APC. I set apc-enable = 0, but still get this when attempting a (still failing) logon

80.15.104.74 - apc [09/Aug/2011:23:40:51 +0200] "GET /fr/poormanscron/run-cron-check HTTP/1.1" 200 20 "http://www.shareontheweb.com/" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"

80.15.104.74 - apc [09/Aug/2011:23:43:16 +0200] "POST /user HTTP/1.1" 200 2612 "http://www.shareontheweb.com/user" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"

With APC disabled, the sites become incredibly slow, and the cpu goes near 100%

Tue, 08/09/2011 - 19:19
Locutus

With APC disabled, the sites become incredibly slow, and the cpu goes near 100%

But can you log in then? For now, top priority would be to find out why you get these odd login problems.

Also, what kind of websites are those that they become so slow when you don't use a PHP cacher? I turned off X-Cache a while back on my system, to be able to use FCGI without nuking my memory, and website performance impact was nowhere near what I feared it would be.

Tue, 08/09/2011 - 19:20
Locutus

With APC disabled, the sites become incredibly slow, and the cpu goes near 100%

But can you log in then? For now, top priority would be to find out why you get these odd login problems.

Also, what kind of websites are those that they become so slow when you don't use a PHP cacher? I turned off X-Cache a while back on my system, to be able to use FCGI without nuking my memory, and website performance impact was nowhere near what I feared it would be.

Tue, 08/09/2011 - 19:20
Locutus

With APC disabled, the sites become incredibly slow, and the cpu goes near 100%

But can you log in then? For now, top priority would be to find out why you get these odd login problems.

Also, what kind of websites are those that they become so slow when you don't use a PHP cacher? I turned off X-Cache a while back on my system, to be able to use FCGI without nuking my memory, and website performance impact was nowhere near what I feared it would be.

Tue, 08/09/2011 - 19:21
Locutus

(Sorry for multiple post, the website was failing again.)

Wed, 08/10/2011 - 01:35
jvieille

No, I still can't login. I get rid of APC and get the server working fine without it.

This is what I get in VM access log for the site I am trying to log in

80.15.104.74 - - [10/Aug/2011:08:21:42 +0200] "POST /user/login HTTP/1.1" 200 5691 "http://web2.see.asso.fr/user/login" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"

No get in return as I should - below a "normal" access log for a login action

80.15.104.74 - - [10/Aug/2011:08:32:50 +0200] "POST /user HTTP/1.1" 302 699 "http://www.b2mml-services.com/user" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"
80.15.104.74 - - [10/Aug/2011:08:32:50 +0200] "GET /user/1 HTTP/1.1" 200 4401 "http://www.b2mml-services.com/user" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1"
Wed, 08/10/2011 - 03:59
Locutus

Okay, sorry, I'm fresh out of ideas here, except for the suggested thorough comparison of config files for Apache and PHP (and possible further used web software like Drupal and PHPMyAdmin).

Wed, 08/10/2011 - 04:18 (Reply to #31)
jvieille

Thanks a lot for having spend that time with my problem.
I scrutinized the running processes on both machines. I found the following differences I couldn't explain by myself:
(the "Bad" one is Debian Lenny, the "Good" one is Ububuntu 10.04 LTS)

1) /sbin/init in Good, init [2] in Bad
2) /usr/bin/dbus-daemon --system in Bad only
3) /USR/SBIN/CRON and /usr/sbin/cron in Bad only
4) /usr/sbin/mysqld in Good, /bin/sh /usr/bin/mysqld_safe and /usr/sbin/mysqld --basedir=/usr --datadir=/home/mysql --user=mysql --open-files- ... in Bad
5) /usr/share/webmin/virtual-server/lookup-domain-daemon.pl in Bad only
6) avahi-daemon: chroot helper, avahi-daemon: running [ks312228.local], -bash in Bad only
7) /usr/sbin/irqbalance, tlsmgr -l -t unix -u -c, upstart-udev-bridge --daemon in Good only

Note: I have just completed a full virus scanning - nothing

Wed, 08/10/2011 - 04:27
Locutus

Another idea. Do you get any entries in /var/log/syslog, /var/log/messages or /var/log/kern.log? If there is a low-level issue with some protection software (apparmor comes to mind) preventing file access, they should get logged there.

Also, please follow my advice and compare the config files of Apache and PHP (and possible further web software) on the "bad" and "good" server. I really can't do more than strongly suggest that.

Wed, 08/10/2011 - 05:14
jvieille

1) Looking at the processes, I noticed that MySQL was not running the same way in both servers:

In bad, the process is

/usr/sbin/mysqld --basedir=/usr --datadir=/home/mysql --user=mysql --open-files-limit=1000 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306

by mysql/mysql

its parent is /bin/sh /usr/bin/mysqld_safe by root / root

its parent is init [2]

in Good, the process is /usr/sbin/mysqld by mysql/mysql

its parent is /sbin/init by root/root

2) I don't see anything in any logs.

3) I have checked everything reported by phpinfo

The only different thing is the cookies generated from the login attempt which is quite big in Bad, most of the time iniexistent in Good

4) I checked Apache over and over again, can't see anything

May be I should consider reinstalling Apache? How could I do that without losing my parameters and Virtualmin control?

Wed, 08/10/2011 - 10:17
jvieille

Just an observation:

It looks like any HTTP GET request is correctly honnored, while POST requests are received by Apache but not further processed. That might be Apache or PHP doing nothing when receiving a POST request

Fri, 08/12/2011 - 00:56
jvieille

Problem solved:

I really don't know how this occurred, but the php variable max_post_size was configured to 2G, but did not behave accordingly. Instead of making php accepting 2 Gig post requests, it made it accept only negative values!

I got this kind warning: [Thu Aug 11 17:04:28 2011] [error] [client 80.15.104.74] PHP Warning: Unknown: POST Content-Length of 177 bytes exceeds the limit of -2147483648 bytes in Unknown on line 0, referer: http://web2.see.asso.fr/admin/settings/performance/authcache

This use to work before, it is very strange that it stopped interpreting correctly this value. The main problem was that I never got this warning before I tweaked php.ini to record errors: by defaut, php do not log anything...

Once I got the message, it was a matter og minute to sovle it... changing 2G by 2000M...

Thank you again very much for your care in this really tricky case!

Fri, 08/12/2011 - 03:44
Locutus

Are you certain you wish PHP to accept 2 gigabytes of POST requests? :) What would you possibly need such large incoming web data transfers for?

Fri, 08/12/2011 - 04:30
jvieille

I wish no limit to post large files (HD videos for example) , I was advised to set this variable at least the value set for file uploads.

Fri, 08/12/2011 - 05:35
Locutus

You're aware that POST requests do not support resuming uploads, right? :) I.e. if you upload an 1.8 GB HD video, and the transfer stops at 1.79 GB, it's wasted.

There are much better suited methods to upload such a large amount of data than HTTP POST.

Fri, 08/12/2011 - 06:23
jvieille

Yes, you are certainly true, and I probably don't need to set this that high. I don't know exactly how Drupal (my main applications framework) handles file uploads, they are probably smart.