What if i only have two machines to use? Don't have the budget right now for "4." I really appreciate your detailed response. This is exactly what I've been trying to implement as far as backup and disaster recovery. I was hoping to see a "hot swap" feature - where ServerA.com was updated to ServerB.com and ServerB.com was an up to date stand by. Is DRBD my best recourse?

DRBD is great for keeping partitions on 2 separate machines in sync. I've heard it described as a kind of "distributed raid array".

I guess this would work well if you used DRBD on your web-content partition. The only drawback is that you can only access one of the DRBD drives at any given time (this might have changed with more recent DRBD releases) - which might be just what you want. Heartbeat can take care of starting apache on the backup server when the primary one goes down.

If it were me, though, I would want to set it up somehow so that you could use both webservers to load balance all the time, and just point them at the "live" content on one server or the other.

The way I would do this is to set up an NFS server on the DRBD content, and mount it as a separate NFS mount point on both servers. (ie. one server would be mounting its own content via NFS - the other would be mounting the opposite's content. The source of this mount would automatically change whenever the other DRBD host became primary, but the content would always be the same)
Then I would point my apache instances to this mount point, and load balance them using ldirectord.

In this way you could take advantage of both apache servers whenever both machines were turned on, and if one ever went down it would automatically fail-over to the remaining server without interruption.

This is kind of what I figured...

In the VirtualMin paradigm though, everything is by user and the apache content directory is a pluralism. Basically, to accurately provide fail over AND load balancing, all the /home/[user_xyz]/public_html directories have to be in sync. (As opposed to /var/www/htdocs/ or something like that) In addition to that, in the virtual hosting model, there are databases and variances in the php.ini configurations per user...and user passwords/id/groups and such are stored elsewhere...meaning that if /home/[user_xyz] is in sync, it does not mean that the users, data, and virtualmin working parts and pointers are all in sync.

I don't have much experience with NFS but I think I understand what you are suggesting. I am using older gear, so my intranet is 100MB and I would assume that this would affect performance in terms of read(s)/write(s) to the NFS mounted volumes? Basically, all but "boot" would need to be replicated...I've never used the load balancing module you mentioned.

Right now - I am just striving to get the Hot Swap to work. DRBD is the soundest means I can think of to replicate all the databases, users, files, and such. Basically, the "/" partition on serverA is the same as the "/" partition on serverB.

This is a simple guide...
http://wiki.centos.org/HowTos/Ha-Drbd

But of course if you did not have the foresight to keep some of your drive partition un-allocated, then you'll have to either use external meta-data for DRBD or "shrink" that partition's file system.

http://www.drbd.org/users-guide/ch-internals.html
(The external metadata option is, um, not so good for recovery.)

..and of course, shrinking a mounted "/" is not exactly fun...

http://www.howtoforge.com/linux_resizing_ext3_partitions

I'll let you know how this adventure plays out. Many VirtualMin users have a configuration similar to the one you have mentioned. I'm a bit leery about using NFS - but I am willing to sacrifice a little performance if it means solid fail-over...

Thanks for your suggestions!

You may be interested in this thread as well...in particular, Jamie's response to this feature request...

http://www.virtualmin.com/forums/virtualmin/clustering-virtualmin.html

I am not sure what you mean by "/etc/apache/sites-available" and "sites-enabled". What is your linux distro? I am using CentOS. There is a /etc/httpd/ folder with config files in there for global and for clients and they are in /etc/httpd/conf/.

Also - are you saying that when you run the synchronization on your front-end webservers that the /home/[user] directories get synced as well? There must be something wrong with my install, I've tried this and found a few bugs.

http://www.virtualmin.com/index.php?option=com_flyspray&Itemid=82&do=details&task_id=4097&Itemid=82&string=&project=1&search_name=&type[0]=&sev[0]=&pri[0]=&due[0]=&reported[0]=&cat[0]=&status[0]=&percent[0]=&opened=merlynx&dev=&closed=&duedatefrom=&duedateto=&changedfrom=&changedto=&openedfrom=&openedto=&closedfrom=&closedto=

Anywho - thanks a ton for posting this. It would be awesome to have a section in the documentation or on the wiki that covers different scenarios of this sort of implementation.

Oh yes, Debian does this with it's apache2 package. They're just basically directories of symlinks to the config files you want to "activate". Sites-enabled is then read by apache.conf (httpd.conf).

/etc/apache2/conf.d
/sites-available
/sites-enabled (symlinks to configs in sites-available)

The way I have things set up, there is no synchronization. Both webservers mount the same web content, stored on a DRBD-based nfs cluster. I guess I forgot to mention that the Developer's server mounts this content under /home, and the webservers mount the same content under /var/www. (this makes absolute paths a bit confusing though)

Content Server A _shared IP --> Mounted on Webserver A __ load balanced
Contnet Server B / --> Also on Webserver B /
--> Also on developer server <-- access via VPN

Brent,

Are you using virtualmin for your hosting environment?

I am kinda confused. In the context of Virtualmin on CentOS there is no "enabled-sites" like you mention in Debian, and I am not sure if I understand how that would paradigm with CentOS's configuration with Apache in the context of Virtualmin handling the /etc/httpd/conf/httpd.conf file and the VirtualHost entries there.

Which machine is virtualmin installed on?

You have two front-end webservers mounting the same content directories from one development server (which is clustered - for failover). That much I think I get.

#1 Development server --mounted to -->Server A
|| --mounted to -->Server B
||
#2 [drbd clustered Dev server]

So if Server A goes down - Server B will "fail over."
If #1 goes down, #2 will fail over.
In both instances, vise versa.

Do you run virtualmin on the Development Server then?

Where do you run bind? What does your bind config look like here? Do you route all FTP requests to the Dev server as well? I notice you put postfix config on the webserver(s) A & B - does this mean that mail is routed through them but ultimately lands in the "Development" server box?

What about usermin and email sending/receiving?

I am just trying to understand how you got all the pieces to fit together...

Thanks for the dialogue - it is very helpful!

CentOS there is no "enabled-sites" like you mention in Debian, and I am not sure if I understand how that would paradigm with CentOS's configuration with Apache in the context of Virtualmin handling the /etc/httpd/conf/httpd.conf file and the VirtualHost entries there.

Virtualmin works with the OS, not separately from it. If the OS keeps things in separate VirtualHost files (as is the case on Debian and Ubuntu), Virtualmin does, too. If the OS keeps everything in one httpd.conf (as is the case on CentOS), Virtualmin does that.

Don't let a minor semantic difference between two systems throw you--they're doing the same thing in slightly different ways.

As for some of your questions, I don't know how Brent does it (and there are many ways, with their own positives and negatives). But, here's how I'd probably do it:

I'm assuming either server can take over the IP(s) from the other, in the event of failure. Everything gets a lot harder if you can't take over the IP.

Where do you run bind?

On both systems. You need two anyway. So, have one slave to the other, and vice versa. This is no different than what the documentation for cluster slave DNS autoconfiguration tells you to do...you just do it twice, once on each machine. DNS is, by far the easiest service to provide redundancy for. Don't let it intimidate you--it's designed from the ground up to be highly redundant. And Webmin takes that one step farther by being able to easily promote a slave zone into a master zone, if you lose your server A and need for server B to become the master for a week while you get server A back into production, or whatever.

What does your bind config look like here?

Follow the autoconfiguration guide for both server A and server B, and you won't need to care. Really don't fret over BIND. It should just work.

I don't know about the other stuff, as I'm not remembering what all this thread was about. But I just wanted to point you in the right direction on BIND, because I could tell you wanted to make it way more complicated than it needs to be. ;-)

I have VirtualMin running on my "Developer's" server. This is the only part that I do not have redundant servers for (because everything on this server can afford to be down for a reboot here and there)

So I have the Development server set up pretty-much as a standalone VirtualMin server, with Virtualmin, Bind, FTP, popa3d, and Postfix running there. Then I export the apache configs from this server to my webservers, and have created a custom /etc/init.d/remote-apache script to start and stop apache on those servers from the Virtualmin server. (All my servers are running Debian Etch)

I think you understand the web content part, so I won't go over that again here.