Cloudmin multi-server design advice please

Hi all

I have a couple of dedicated servers at present and they are both configured as KVM hosts with one of them also being the Cloudmin Master. The servers are in the same rack at the DC, and are connected via 4 x gigabit NICs on each server through a gigabit switch. Effectively they have a ~4000mbit internal network, with 100mbit uplinks to the internet.

What I'm trying to do is adjust the design of this arrangement slightly. It would make more sense if the Cloudmin Master was located on a different machine - ie not one of the host machines. If one of these machines fails, there's scope for it to take over the virtual machines from the other. However, if it's the Cloudmin Master that fails, then I lose control for a while until I get another system up and running.

My question is related to bandwidth usage and data transfer etc. Obviously the Cloudmin Master handles the backup processes, and I have full system backups from each host machine running to the other host (so that if one fails I can recover from the remaining machine). Assuming that the Cloudmin Master is not within our fast LAN and actually in another DC entirely, what speed limitations should I experience when: a) backing up from one host to the other? b) recovering from one host to the other? c) creating new virtual machines? d) saving images ready for creating a new VM? e) moving VMs from one host to another? f) anything else that I may not have considered?

Any help, advice or past experiences would be greatly appreciated.