I’m not sure if there’s any performance boost in switching to this new virtual hardware? – VMware only lists added Windows Cluster support as a new feature. But I’m the type of guy who is bugged by having Windows 2008 templates which doesn’t match the default hardware, so I set out on the quest to upgrade the SCSI controller of my 2008-templates…

It’s not quite as simple as it might sound – you can’t just switch the controller as Windows lacks the driver to use the new controller, so this operation renders the OS unbootable. (until you switch the controller back)

Before you try to use the described procedure, be sure to have the VMware Tools in the VM up-to-date and upgrade the virtual hardware to version 7!

The idea behind this procedure is:

1. Add a temporary IDE hard disk (VMware allows you to have SCSI and IDE drives simultanious, but not two SCSI drives with different controllers).
2. Let Windows install the IDE-drivers.
3. Remove the temporary hard disk.
4. Switch the active hard disk controller to an IDE controller.
5. Add a temporary SCSI hard disk with the new controller type.
6. Let Windows install the new SCSI-drivers.
7. Remove the temporary hard disk.
8. Switch the active hard disk controller back to a SCSI-controller.
9. Clean up unused drivers.

If you want the detailed instructions on how to perform the operation, read on below.

I’m sure there’s an easier way to install the new drivers before switching the SCSI-controller, but I’m no Windows wizard, so I prefer to let Windows discover the new hardware by itself.

This procedure should work on other Windows versions as well, but I’ve not tried it on anything besides Windows Server 2008.

Needless to say: make sure you have a backup of your VM before something is screwed up!

Detailed instructions.

Power down the VM or convert it to a virtual machine if it is a template.

Add an IDE hard disk:

Boot the server and let Windows install the IDE drivers (answer “I moved it” if asked). If you wan’t you can see the two disk drives in the Device manager:

Shut down the server.

Delete the IDE drive again (remove the files from disk):

Also remove the SCSI hard drive (make sure NOT to delete the files from disk!).

Log into the Service Console and find the .vmdk file defining the hard disk. Edit this file with your favorite Linux text editor (“vi” for instance).

Change the adapter type in the line:

ddb.adapterType = “lsilogic”

to

ddb.adapterType = “ide”

Add a new hard disk, choose to use an existing virtual disk:

Browse to the harddisk file, and make sure it is recognized as an IDE disk:

Add another new hard disk, this time create a new virtual disk:

Make the new drive a SCSI disk:

This will get assigned a “LSI Logic SAS” controller if it’s Windows 2008:

Boot the server, log in and let Windows install the new SCSI driver.

Shut down the server and delete the new SCSI-drive again and remove the “old” IDE disk retaining the file:

Edit the .vmdk file in the service console, and the adapter type back to “lsilogic”:

ddb.adapterType = “lsilogic”

Add a new hard disk, choose to use an existing virtual disk. Browse to the harddisk file.

It will now recognise it as a SCSI drive:

And it will use the new SAS controller (if it’s Windows 2008):

Boot the server and check that everything work as expected.

If you want to clean up the now unused drivers, start a command prompt with admin rights (Run as administrator):

Type “set devmgr_show_nonpresent_devices=1″ at the prompt.

Launch the Device Manager from the prompt with the “devmgmt.msc” command:

Select “Show hidden devices”:

Find the non present devices that are grayed out, and delete them (the old “VMware Virtual disk SCSI Disk Device”, the “IDE Hard Drive ATA Device” and the old “LSI Adapter, Ultra 320 SCSI 2000 series”):

Now everything looks neat and clean:

If it original was a template, shut the server down and convert it back to a template.

As an alternative I’ve been exploring NetApp’s VIBE (Virtual Infrastructure Backup Engine) script, and it actually seems to be doing the job just right!

VIBE makes VMware snapshots of all the VM’s on the datastore, just like SnapManager, but it does so one VM at the time – it take quite a bit longer to generate (and remove) all the snapshots, but gone are all the timeout problems.

The idea is that SnapManager makes VMware snapshots of the running machines, then does a NetApp Snapshot of the storage volume and finally deletes the VMware snapshots – this is done to be sure that the virtual harddrives are in a consistent state when the NetApp Snapshot is made. If the applications running in the virtual machines supports VSS (Volume Shadow Copy) it’ll even be application consistent!

The after NetApp Snapshot are created it is SnapMirrored (copied) to a Filer in another physical location for disaster recovery purposes.

It sounds great, and it even works great then I do a backup of a single VM. But, yes as the title suggests there is a “but”, when I try to backup my entire datastore, SnapManager tells Virtual Center to make snapshots of the all the running virtual machines at once. It might be that our NetApp Filer is too slow, or due to some other bottlenecks in our setup, but when VC tries to take 26 simultaneous snapshots a few (between 2 to 7) of them randomly suffers a “timeout”.

The failed snapshots makes SnapManager abort the backup.

I’ve looked high and low for a setting to either:

• Make SnapManager only do a few snapshots at a time,
• Extend the snapshot timeout value in VC (it seems to be 20 seconds) or
• Make VC queue the snapshot commands so that only a few are executed in parallel.

But I havent been able to do either so far…

I also experienced two cases where the .vmx file of powered-off VMs mysteriously got deleted somewhere in the process… Bad karma!

So, until I got things sorted out, we’re back to plain ol’ NetApp Snapshots for the backup of our virtual machines. (of cause combined with client based backup of important data inside the VMs).

All in all we are quite happy with this solution, but as our storage capacity is running a bit low, I was extremely dissatisfied to find that moving virtual machines from storage to storage, would convert our thin provisioned disks to thick disks (a 30 gig harddrive with 10 gig data would fill 30 gig on the storage and not 10 gig).

As our Netapp filers are too old and slow to run deduplication I tried to use the “Shrink” function from the VMware Toolbox inside the virtual machines, hoping that this would convert the disks back to thin disks. But no luck there; this action actually expands the virtual disk to fill the maximum possible if it wasn’t already doing so!

I’m sure this behavior is not intended from VMware, and it will probably be fixed sometime in the future. But until then, I discovered a way to convert the disks back to thin disks while the virtual machines are running.

I take no responsibility for this procedure, and if you lose your virtual disks, don’t come blaming me, but roll back to your backup.

Here we go:

First of all, to get the most out of shrinking your disk, you should do the following inside the virtual machines.

1. Defragment your virtual disk

Use the build-in defrag-tool in Windows or a third part tool of your choice.

2. Shrink the disk using VMware tools inside the virtual machine

Go ahead and open the VMware toolbox and find the “Shrink” function

When you click the “Prepare to shrink” you start a two stage process; first the application fills all vacant space on the harddrive with large empty files

This process will take quite some time if you have a lot of unused space on your disk. If you look at the root of the harddrive, you’ll find that it simply generates a number of 2 gig files named “wiper0″, “wiper1″ etc..

These files are simply used to overwrite any “deleted” files on the disk; when Windows deletes a file, it doesn’t actually remove the data, it just removes any reference to it – therefore it still takes up space on your disk. The “wiper”-files allow ESX to identify truly unused blocks on your disk.

The next step the Toolbox is to supposed to do, is to tell the ESX host to do the Shrink

But unfortunately this is never done. Hence we need to do some additional work:

3. Take a snapshot of the virtual disk

This is done to have a static non-changing disk file to work on. When a snapshot is active, every change to the harddrive is written into a separate delta-file.

4. Open a session to an ESX host with access to the storage

Either use a SSH client or access the service console directly – make sure to be “root” before you try to proceed.

Change directory to where the virtual machines stores its files:

cd /vmfs/volumes/netapp10/Win2003helper/

In this example my storage is named “netapp10″ and the virtual machine is called “Win2003helper”.

A simple “ls -l” command reveals which files exists, the interesting ones are:

-rw-------    1 root     root     16795648 Oct  7 13:15 Win2003helper-000001-delta.vmdk
-rw-------    1 root     root          235 Oct  7 13:14 Win2003helper-000001.vmdk
-rw-------    1 root     root     8589934592 Oct  7 13:14 Win2003helper-flat.vmdk
-rw-------    1 root     root        19395 Oct  7 13:14 Win2003helper-Snapshot1.vmsn
-rw-------    1 root     root          406 Oct  7 12:59 Win2003helper.vmdk

The file “Win2003helper.vmdk” contains metadata about the actual harddisk file, which is named “Win2003helper-flat.vmdk”. As you can see we also have a snapshot in place; the .vmsn file contains metadata about the snapshot, and the aforementioned delta disk file is named “Win2003helper-000001-delta.vmdk” (with a matching metafile).

If you’re the curios type, you can do a “cat <filename>” to check out the content of meta files.

5. Make a “thin” clone of the snapshot

Now it’s time to do the magic! Use the “vmkfstool” command to make a clone of the virtual harddrive:

vmkfstools -i Win2003helper.vmdk -d thin Win2003helper_thin.vmdk

The “-d” option tells vmkfstools to make it a thin provisioned disk. The clone will be made from the non-changing snapshot (remember that changes are written to the delta-disk), so we don’t get into trouble with inconsistent drives.

Destination disk format: VMFS thin-provisioned
Cloning disk 'Win2003helper.vmdk'...
Clone: 100% done.

Wait for the cloning to finish. When it’s done, a “ls -l” will reveal the new disk:

-rw-------    1 root     root     8589934592 Oct  7 13:24 Win2003helper_thin-flat.vmdk
-rw-------    1 root     root          437 Oct  7 13:24 Win2003helper_thin.vmdk

You might be surprised to see that the file size is exactly the same at the bloated disk, you were hoping to shrink – but don’t panic, this is only due to the fact that the service console sees the size of the drive, and not how much space it occupies on the storage. Go ahead and browse the storage from the VMware Infrastructure Client, and you should see the true picture

6. Replace the thick disk with the thin clone

Go ahead and overwrite the thick disk with the clone just created:

mv -f Win2003helper_thin-flat.vmdk Win2003helper-flat.vmdk

And then remove the obsolete meta file “Win2003helper_thin.vmdk” either with the Datastore Browser or from the command line:

rm -f Win2003helper_thin.vmdk

7. Delete the snapshot

Finally you can delete the snapshot you created in the Snapshot Manager

Just press “Delete All”, and it will apply all changes that might have happened to the harddrive while we were cloning the disk.

8. Enjoy

If you take a look at the Datastore Browser, you will now find everything cleaned up, and the virtual disk shrunk to fit the actual usage in the virtual machine.

Of cause the file will grow as you add data to your virtual disk, and if you delete a lot of data, you might consider going thru the procedure again, to reclaim that space.