Wednesday, November 04, 2009

VMWare Storage VMotion Plugin (SVMotion) Gotcha!

We use VMWare Virtual infrastructure extensively at work and in the 4 years we’ve been running our core infrastructure on it it has saved us thousands in time and effort. It constantly amazes me just how clever and technologically advanced it is.

However, when it goes wrong, there’s a LOT more at stake. More than once, the management GUI sits there doing “something”, with a progress bar slowly creeping across a screen and the whole infrastructure is doing something unexpected. You feel quite helpless.

The most recent of these was yesterday. We were preparing for a major database upgrade so was checking we had enough storage space to snapshot the SQL servers (a topic worthy of many a blog post!). When I noticed I didn’t have enough space on one of the storage groups I though I would use storage VMotion to move some of the drives and the main VMWare config file onto a different datastore. VMotion is the “killer app” in VMWare and allows you to move a running Virtual Machine around onto different hardware without losing any service. The storage VMotion moves virtual drives around on the SAN, again without losing service. We’ve used a free plug-in (http://sourceforge.net/projects/vip-svmotion/) that puts a GUI onto the SVMotion tools a few times to just move drives around on the SAN and it works a treat. You can balance storage use on the SAN with zero down time.

However, yesterday I had to move a VMX (VMWare config) file as well. I had to do this as VMWare by default stores snapshot delta files in the same location as the VMX file. There are ways of forcing the snapshot files to be written elsewhere but the thought of just running an SVMotion to move the VMX file seemed easier….at the time.

Within the storage VMotion GUI you would typically just drag and drop and virtual drive file onto a new datastore and click apply. I assumed moving the config would be the same, it’s NOT. When you drag the VMWare config onto a new datastore, it automatically snaps ALL virtual drives onto the same store. For a small machine with everything on one store that may not be a problem. For a 1/2TB SQL server with 5 drives spread across 4 datastores it was a BIG problem, that I didn’t spot until after I clicked apply!

I took me about 10 minutes to work out what sort of problem I’d kicked off and then a further 30 minutes to work out why. I had no choice, I had to make sure my target datastore had 1/2TB of free space by moving other VMs off. I just had to let it run as by all accounts trying to fix a failed storage VMotion is a much worse prospect.

With the benefit of hindsight, this is what you MUST do if you just want to move a VMX (Config) file.

  1. Open up the SVMotion GUI and make a note of the current drive/datastore locations for your VM.
  2. Drag your VMWare config file to the new target datastore (It will fail if there isn’t enough space to house the complete virtual machine – I didn’t know that yesterday). You should see the complete VM tree on the new datastore.
  3. This is the IMPORTANT bit. Drag the individual disks BACK to their original datastores.
  4. Click apply. VMWare should then just move only the files and drives you want to move onto the target datastore.

Fair play to VMWare, although it took over 8 hours to SVMotion a 1/2TB SQL server that was being hammered in the middle of a working day, the server didn’t miss a beat. No slowdowns, no complaints from users, nothing. Phew!

No comments:

Post a Comment