Submitted by lulatsch66 on Wed, 11/28/2012 - 02:52
Howdy,
the recent change from "FollowSymLinks" to "SymLinksifOwnerMatch" broke all of our vhosts.
Why? We use shared sources at e.g. /var/lib/typo3/... so these sources are owned by root, the typo3_src in public_html is a symlink to e.g. /var/lib/typo3/4.5, which again points to the actual release, making (security) updates much easier. IMHO this is a usual practice.
Using fcgid and a dir mask of 750 in /home/* I see no reason why FollowSymLinks could imply a security problem here?
How can we prevent such changes for existing vhosts in the future, and how can we update other server without breaking all vhosts? Is there any setting which can disable these changes when upgrading?
Thanx for your support!
Falko
Status:
Closed (fixed)
Comments
Submitted by andreychek on Wed, 11/28/2012 - 08:44 Comment #1
Howdy -- although Apache executes PHP and CGI scripts as a given user, when it accesses static HTML, CSS, images, and the like, it does so as the Apache user.
With a permission scheme of 750 on all the dirs in /home, Apache can't actually read any of those files.
To solve that, Apache is added to each Virtual Server owner's group, as you can see in the /etc/groups file.
Unfortunately, we recently discovered that, using those privileges and with FollowSymlinks active, it's possible for a given user to trick Apache into reading the files of users.
By changing FollowSymlinks to SymLinksifOwnerMatch, that resolves a rather large security hole.
I don't know of a setting to prevent updates to the config, Jamie could comment on that a little later today.
Submitted by JamieCameron on Wed, 11/28/2012 - 11:47 Comment #2
The best work-around here would be to put a
.htaccessfile in each domain's directory that contains :Options +FollowSymLinksSubmitted by lulatsch66 on Wed, 11/28/2012 - 12:15 Comment #3
Thanx for your answer, but .htaccess are disabled for most vhosts. It's not even possible to have symlinks to dirs owned by root.
Why is where no global setting for virtualmin to avoid such behaviour?
If the update was done already: in which situations will virtualmin repeat this behaviour of changing FollowSymLinks to SymlinkIfOwnerMatch? Is it enough to change all SymlinkIfOwnerMatch entries to FollowSymLinks in apache vhosts and virtualmin templates once, must we change this every time virtualmin-server will be updated?
Thanx again for your support.
Submitted by lulatsch66 on Wed, 11/28/2012 - 12:29 Comment #4
ok, I understand that the user www-data must have access to the virtual server file directory. But isn't the fcgid process running as the server owner using apache suexec to prevent access to other user's files?
Is there another way how we could share common directories owned by root between vhosts?
Thanx for your answer.
Submitted by andreychek on Wed, 11/28/2012 - 14:53 Comment #5
When using suexec through FCGID and CGI, that only processes PHP/CGI scripts.
Other files, such as static HTML/CSS/images, continue to be processed directly by Apache.
Because of that, it's possible to trick Apache into doing very bad things when using file types other than PHP/CGI.
So how might you share common directories?
Well, that all depends on the application in question... it shouldn't be necessary to use a symlink though.
It may just be a matter of specifying the full path to the location of your common path in your application's config, rather than using a symlink.
You may also be able to add that common path to the PHP "include_path" directive. That way, when PHP attempts to load the script, it consults with the include_path to find the .php files.
Submitted by sgrayban on Thu, 11/29/2012 - 03:43 Comment #6
However implementing this without warning is a huge mistake as a lot of people like me and my clients do the same thing with typo3 and other programs.
The impact this has caused downtime, broken sites and lost sales and a shit load of very unhappy users.
Not only was money lost I also had to charge them to fix what YOU broke without consulting anyone here first.
This is the first time I am really pissed off at all of you for assuming what you think best is what we have to do without so much as a warning. This is the type of crap cPanel developers do, lets not start down that path.
Rollback that symlink enforcement ASAP.
Submitted by sgrayban on Thu, 11/29/2012 - 03:49 Comment #7
For those of that have broken sites you can run the following command where your conf file(s) are located at.
For debian the files are located at /etc/apache2/sites-enabled
sed -i 's/SymLinksifOwnerMatch/FollowSymLinks/g' *.confSubmitted by helpmin on Thu, 11/29/2012 - 15:29 Comment #8
@sgrayban Not sure why you didn't test an upgrade in a sandbox first (which is exactly the reason why you have two licenses for virtualmin). Every sys admin should always do this to avoid "downtime, broken sites and lost sales and a shit load of very unhappy users" ("due diligence").
Even though the virtualmin folks have produced amazingly stable updates in the last couple of years, no software is perfect :-)
Submitted by sgrayban on Thu, 11/29/2012 - 18:55 Comment #9
I never had a reason to test any update until this one because Joe and Jamie always promoted VM as a non-destructive control panel but as we see now they have forgotten that promise and I have disabled all package updates for all my clients.
This little stunt is going to cost my clients more money because now I have to check every release from head to toe before I allow it... and if I find something that will break sites again I'll have to bitch to get a fix when all of this could have been avoided by sending out a email that their was a security issue that will be implemented in the next release and to check it out before upgrading.
Seriously this is the first time I have seen a arbitrary change like this without proper notice. I had no reason to not trust Joe or Jamie until now.
What is done is done but a heads up would have been nice... I suggest a mailing list just for things like this.
Submitted by lulatsch66 on Mon, 12/03/2012 - 14:55 Comment #10
@andreychek: I see no way how to accomplish this - to use a common path in the application's config, as the application itself is symlinked.
Just for example, usually it's done like this:
:~# cd /var/lib/typo3/:/var/lib/typo3# ls -l
insgesamt 96K
lrwxrwxrwx 1 root root 16 17. Nov 17:30 4.5 -> typo3_src-4.5.22
lrwxrwxrwx 1 root root 15 17. Nov 17:30 4.7 -> typo3_src-4.7.7
lrwxrwxrwx 1 root root 3 25. Mär 2011 default -> 4.5
drwxr-xr-x 4 root root 4,0K 8. Nov 12:50 typo3_src-4.5.21
drwxr-xr-x 4 root root 4,0K 12. Nov 22:56 typo3_src-4.5.22
drwxr-xr-x 4 root root 4,0K 8. Nov 13:11 typo3_src-4.7.6
drwxr-xr-x 4 root root 4,0K 12. Nov 23:18 typo3_src-4.7.7
and in the vhost's public_html dirs:
lrwxrwxrwx 1 falko.trojahn trojahn.de 19 27. Jun 2010 index.php -> typo3_src/index.php-rw-rw---- 1 falko.trojahn trojahn.de 71 5. Sep 09:01 robots.txt
lrwxrwxrwx 1 falko.trojahn trojahn.de 15 27. Jun 2010 t3lib -> typo3_src/t3lib
lrwxrwxrwx 1 falko.trojahn trojahn.de 15 27. Jun 2010 typo3 -> typo3_src/typo3
drwxrwx--- 4 falko.trojahn trojahn.de 4,0K 2. Dez 11:41 typo3conf
lrwxrwxrwx 1 root root 18 29. Nov 17:53 typo3_src -> /var/lib/typo3/4.5
I'd like to know the answers to the following questions, yet:
Could where be a global or per-vhost-setting for virtualmin to disable the check for "FollowSymLinks"?
If the update was done already: in which situations will virtualmin repeat this behaviour of changing FollowSymLinks to SymlinkIfOwnerMatch? Is it enough to change all SymlinkIfOwnerMatch entries to FollowSymLinks in apache vhosts and virtualmin templates once, or do we have to deal with these changes this every time virtualmin-server will be updated?
Thanx in advance! Falko
Submitted by JamieCameron on Mon, 12/03/2012 - 19:00 Comment #11
The next Virtualmin release will give you a choice of fixing FollowSymLinks or not. However, it will be a global setting .. not per-domain.
Submitted by lulatsch66 on Tue, 12/04/2012 - 01:08 Comment #12
That's great news - thank you very much.
Yesterday we came across another problem that seems related to this update - but I'll have to discover further.
Adding new subdomains vhosts gave us an "VirtualHost *:80" entry instead of "VirtualHost IP-Address:80", so the vhost wasn't accessible. Even going to the IP Adress module of the vhost and selecting "Shared IP Address" didn't work, had do change manually in site-enabled/xxx.conf.
Thanx again for you support
Falko
Submitted by JamieCameron on Tue, 12/04/2012 - 16:01 Comment #13
The latest update didn't change anything related to the virtualhost line..
Submitted by lulatsch66 on Wed, 12/12/2012 - 17:10 Comment #14
Thanx for the possibility to disable the followsymlink check in 3.97.
Hey,
I hate to say this, but if people took a moment to read the "Change Log" before updating, they'd have noticed that it read for version 3.96:
Hmm, for those who aren't aware, even before the news is published on Virtualmin.com, the change log for Virtualmin will often be published on Webmin.com at:
http://www.webmin.com/vchanges.html
We monitor these change logs all the time to see what's new, or changed as not to have the type of situation affect our cluster of Virtualmin powered machines.
-Peter
*** BTW, Webmin/Virtualmin is NOT destructive. Because the update noted was intended to FIX a security THREAT, I wouldn't consider this a betrayal but more of a blessing. If you have configurations relying on a certain feature or setting, it is YOUR responsibility to read over the changes before applying an update so that you can prepare for such changes. ***
Submitted by Locutus on Thu, 12/13/2012 - 04:57 Comment #16
In an ideal world, everyone would read the change log, and everyone would precisely understand what the possible repercussions of changing "FollowSymlinks" to "SymlinksIfOwnerMatch" are. In an ideal world, everyone would therefore do some testing with existing domains, if they run with the new settings. The change log would, in an ideal world, also list precisely how the Apache directives get changed, to make such tests possible.
Unfortunately, we don't live in an ideal world.
Also, this is the first time I can remember that Virtualmin actually performed invasive changes of this magnitude to existing servers. (Not saying here that the change is not useful!) So people probably just were not prepared for that, and silently assumed that reading the changes log is not strictly required. I sure would have been one of those, if I didn't usually wait about a week before updating my production sites, to see if there are any problem reports. :)