Initial thoughts and feedback with Cloudmin & Xen guests

11 posts / 0 new
Last post
#1 Fri, 06/26/2009 - 12:20
fatbox

Initial thoughts and feedback with Cloudmin & Xen guests

Hi,

I've been playing around with Xen images and wanted to share my experiences, some issues I've run into & some feature requests for Cloudmin.

Some background info on my dom0: - Intel Xeon's - Kernel 2.6.29 with rebased OpenSUSE Xen patches - Xen 3.4.0 - x86_64 - Gentoo (Funtoo)

Kernel & Initrd

When I build my kernels I build two different kernels, one for dom0 & one for domU's.

The way that kernel & initrd images are handled doesn't work too well outside of CentOS/Debian (from reading the code). I don't use initrd's, I have a homogenous hardware environment so my required drivers are compiled into the kernel and I use modules for things like USB devices, IPTables/LVS, etc. As such when I tried to setup the Xen dom0 system in cloudmin it bombed out trying to create an initrd initially (mkinitrd wasn't even installed on my system), once I installed mkinitrd it bombed out because the xenblk module isn't available (because it's compiled in) so I poked around in xen-type-lib, created an initrd by hand and put it in /boot as the filename that cloudmin was looking for.

I would really love to see some better options for handling Kernels & Initrd on Xen dom0's. Maybe enumerate /boot on the dom0 and allow the user to pick a kernel & initrd for each guest on creation as opposed to tying it to dom0. This would also allow a dom0 to contain multiple domU kernels to be used.

On the same topic, pygrub support would nice to see in the future so that domU's can boot of their own kernels.

Network Interfaces

On my Xen dom0 I have two bridges, one carries production traffic the other carries traffic to my NAS. When I create an image with Cloudmin I can only specify the config for one interface. I manually edited the config file and added a second instance to the vif statement with the correct bridge tags. Looking through the options I'm guessing at this point in time my best bet is to use the post modification scripts to set this up for new images. Any chance we could get some more flexible network config options built into Cloudmin?

One thing I would really love to see is the addition of tying IP blocks defined on a dom0 to a particular bridge, thus I could auto allocate from each block for each device.

Xen Config Templates / SCSI device ids

When Cloudmin creates a Xen system it uses 'sdaX' as the device IDs. This causes the guest in my environment to display the ever annoying "XENBUS: waiting for devices to initialize" message and then spend 5 minutes counting down to 0 at which point it boots normally. I can work around this by disabling SCSI support in my kernel but due to option dependancy this is less than desirable.

Changing the device ID for the guest from 'sdaX' to 'xvdaX' skips the waiting message and the guest boots like normal. Like the network interfaces it's possible to edit the config file (and hosts fstab) by hand afterwards, I could also built this into a post modification script, but it seems to make sense to either have this as a configurable option in Cloudmin or at least provide an easy way to edit a template from which the Guest configs are generated so I don't have to go mucking around in xen-type-lib.pl.

Cloudmin system as Xen guest

My Cloudmin install is running inside a Xen guest on a dom0 I wish to manage. When I try to "Find Xen instances" I get the message: The system X.fatbox.ca is already using IP address X.X.X.X

Not a huge deal but it would be nice if Cloudmin could recognize that it's inside a guest and allow for further management (reboots, resources, etc).

SSH Host Key's on repeat installs

If you create a guest, have it come up and generate SSH host keys, delete it and then re-create a new guest that is assigned the same IP you get an error in Last Status about SSH and the detailed status shows all '@@@...' because SSH is returning the error about the host keys changing. When a system is deleted Cloudmin should purge the SSH keys for it from ~root/.ssh/known_hosts

System image caching on dom0's

Most of the dom0's I'm managing are in the same data center as the cloudmin instance on a gigabit network so moving images around when creating instances isn't a big deal, but one of the systems I have under the control of cloudmin is back at my office and while it has a 10mb pipe (100mb out of the DC) the transfer still takes some time and eats up bandwidth (contributing to my 95th percentile). I would love to see some type of caching so that once I've transfered an image it will remain on the dom0 system until the image is updated on the cloudmin system.

I'll be continuing to play with cloudmin over the weekend and will post anything else I come across in this topic.

So far I'm quite pleased with cloudmin. It's shaping up to be a great tool :)

Fri, 06/26/2009 - 15:03
JamieCameron

Thanks for your comments.. my responses to each are below :

Kernel & Initrd

That's a good suggestion, I will look into making the kernel choice more flexible at instance creation time.

Network Interfaces

This is already on my todo list, as a few people have asked about it. The 2.9 Cloudmin release should support multiple interfaces..

Xen Config Templates / SCSI device ids

I didn't know about xvdaX .. probably because I've never seen this error on my test systems. Is there any down-side to using xvdaX all the time, for all systems?

Cloudmin system as Xen guest

You should be able to tell Cloudmin not to try to manage its own system directly by clicking on the Module Config link under Cloudmin Settings, and changing 'Include this system in managed list?' to 'No'. Detection should then work the way you want.

SSH Host Key's on repeat installs

Cloudmin should actually be doing this already, for the root user on the master system. When you delete a virtual system, it's keys should get removed from ~root/.ssh/known_hosts file. Are you still seeing those keys in there?

System image caching on dom0's

That's an excellent suggestion - I will add this in the next release or two.

''

Sat, 06/27/2009 - 00:35
fatbox

Hey Jamie,

Here's a link to more info on the whole xvd namespace. There should be no downside to using xvdX for all images, I'm using it on some Debian Lenny systems, however I've never used CentOS so it should probably be checked there too just in case.

http://lists.xensource.com/archives/html/xen-users/2008-01/msg00850.html

I will do some more testing with the SSH host keys to see if I can come up with a repeatable process. Today was the first time I deleted an image and re-allocated it's address to a new host.

-Evan

Sat, 06/27/2009 - 02:53
MACscr

forget pygrub, we actually want pvgrub (http://wiki.xensource.com/xenwiki/PvGrub), which enables more flexibility in the kernel offerings, plus is a bit more secure. Good suggestions though. Im glad a majority of them are already in the works.

Sat, 06/27/2009 - 19:59 (Reply to #4)
fatbox

Ahh, cool. I actually hadn't gotten around to playing with booting domU kernels directly and hadn't seen reference to pvgrub. I find the Xen documentation WAY behind where they actually are at.

Sat, 06/27/2009 - 20:07 (Reply to #5)
MACscr

When it comes to Xen, like many other projects, its all about their mailing lists and IRC channel. Thats where i get most of my knowledge of it. The docs are useful as well, but as mentioned, usually out of date.

Wed, 10/14/2009 - 15:43
MACscr

Jamie,

Can you explain to me again why we cant use pv-grub, etc, at this point with cloudmin? There were some limitations that you mentioned that would come into play though when allowing guests to run their own kernel. Care to elaborate here?

Wed, 10/14/2009 - 16:54 (Reply to #7)
JamieCameron

Can pv-grub be used even when LVs or disk images are mapped to a single partition in the Xen guest, rather than a whole disk image? Last time I looked into this (which was many months ago), booting a kernel from within the guest required use of whole-disk images, which meant they couldn't be mounted as filesystems on the host system.

But if that isn't the case and it is just a matter of putting bootloader = /path/to/pygrub in the .cfg file, I could definately support it.

''

Wed, 10/14/2009 - 19:48 (Reply to #8)
MACscr

pv-grub looks a bit like this:

kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz" extra = "(hd0,0)/grub/menu.lst"

pygrub looks like this:

bootloader = "/usr/bin/pygrub" extra = "(hd0,0)/grub/menu.lst"

IRC conversation: (7:22:55 PM) MACscr: Can pv-grub be used even when LVs or disk images are mapped to a single partition in the Xen guest, rather than a whole disk image? Last time I looked into this (which was many months ago), booting a kernel from within the guest required use of whole-disk images, which meant they couldn't be mounted as filesystems on the host system. (7:36:37 PM) Kevin: MACscr: I suggest you test that again, last I tried fully that was correct, but apparently pv-grub CAN load files from full-disk filesystems, in another random test (7:36:54 PM) Kevin: recently i've been putting /boot on a seperate disk, for that

Not sure if any info here could help: http://www.linode.com/wiki/index.php/PV-GRUB

I will see if i can dig up any more info.

Wed, 10/14/2009 - 22:39 (Reply to #9)
JamieCameron

Cool, if they can work with partition images then I will definately add support.

Are there any advantages of pv-grub over pygrub, or vica-versa?

''

Wed, 10/14/2009 - 22:54
MACscr

Yes, there are advantages of pv-grub over pygrub, which i listed earlier: http://www.virtualmin.com/node/10273#comment-44519

I dont have much more to add to it except about every article i found comparing them preferred pv-grub and said it was more secure as well.

I think allowing clients pick their own kernels solves a lot of issues and would also remove some of the errors were seeing at boot. Plus it makes the client feel more like they have a dedicated environment to themselves, which they do.

Topic locked