No r/w access on VM after live migration of iscsi hosted xen-domU

Hello

When I try to live-migrate a xen vm through cloudmin the vm looses write access to it's disk (LVM on iscsi). First everything works like expected. Cloudmin does move the .cfg file to the new host, creates the iscsi connection and I can ping and ssh to the VM on the new host after migration. Xentop shows the machine running on the new host. But if I ssh to the vm there's no more write access to the disk. I have to destroy the vm and then I'm able to reboot the vm. Sometimes with, sometimes without the need to first fsck the vm's disk.

If I shutdown the vm before moving, everthing's fine.

BUT: If I have a running vm on host-1 and do a live migration on console directly on the xen host the vm is moved to host-2, still accessible and if I ssh to the moved vm there's still write access to it's disk. I'm not sure yet (would have to test again properly) but I think that I'm only able to live-migrate the vm like this after the new host has an already connected iscsi connection to the iscsi server.

So also here (like in my post some minutes ago) I'm not sure If I have an missconfiguration of the kind the xen-hosts are connecting to the iscsi server. But I do not really know how the iscsi passwords which cloudmin is setting are distributed to the different vm-hosts (if they really are).

Kind regards

Fix

Status: 
Active

Comments

Could you post the output from the move process as shown in Cloudmin when this happens? The code path uses for live migration when disks are on iSCSI is different from the method used for non-live migration, and I'd like to see what order operations are happening in.

Hi Jamie

Here's the output of cloudmin when I live-migrate the vm test0123

Host 1: a.xy.server Host 2: b.xy.server iscsi: i.xy.server

----- Quote moving test0123 from a.xy.server to b.xy.server -------------

test0123 Moving Xen system from a.xy.server to b.xy.server .. Checking if live migration is possible .. .. live migration appears to be possible Adding iSCSI connection to i.xy.server target test0123_1_img on new host system .. .. done

Adding iSCSI connection to i.xy.server target test0123_0_img on new host system .. .. done

Live migrating virtual system .. .. done

Fixing and copying Xen config file /xen/test0123.cfg .. .. copied to /xen/test0123.cfg

Configuring Xen instance for VNC console access .. .. added on dynamic port

Deleting 1 config files from original host .. .. done

Removing iSCSI connection to i.xy.server target test0123_1_img on old host system .. .. done

Removing iSCSI connection to i.xy.server target test0123_0_img on old host system .. .. done

Re-fetching system status .. .. done. New status is : Webmin

Refreshing status of host system a.xy.server .. .. done. New status is : Webmin

Refreshing status of host system b.xy.server .. .. done. New status is : Webmin

.. done

------ End quote moving server ------------

And here's a output of 'xm console test01234' on new host after moving

------ quote output on xm console test01234 after moving ---------

end_request: I/O error, dev sda1, sector 2518800 Buffer I/O error on device sda1, logical block 314850 lost page write due to I/O error on sda1 Aborting journal on device sda1. ext3_abort called. EXT3-fs error (device sda1): ext3_journal_start_sb: Detected aborted journal Remounting filesystem read-only end_request: I/O error, dev sda1, sector 5088 Buffer I/O error on device sda1, logical block 636 lost page write due to I/O error on sda1 journal commit I/O error

------- End quote output on xm console test01234 after moving ---------

Regards

Fix

It looks like maybe even though Cloudmin makes the iSCSI connection on the new system, it isn't actually usable right away.

What gets logged to /var/log/messages on the new host when you do a move?