The WOW Factor

I remember the first editions of VMworld: technologies like vmotion and FT made you just standing still thinking "WOW!".
That "WOW factor" had faded away in later editions: virtualization technology is now a consolidated thing and I've not seen anything really breathtaking.
The last two editions were really all about the cloud (yawn!) and just some other little incremental improvements.
Finally, I saw something that could bring a revolution in the way virtualization clusters are built:
SimpliVity Omnicube and Nutanix Complete Cluster.
It's WOW time again...

VMWare Memory Overhead

While sizing ESX hosts you must include the RAM ESX itself uses for VM book keeping.
Here's a table of Memory Overhead for tipical VM sizes:

VM Size	1G	2G	4G	8G
1cpu	134.80	150.89	183.06	247.39
2cpu	173.29	197.40	245.62	342.05
4cpu	214.05	238.16	286.38	382.80

All values are in MB... or should I say MiB?

Expanding the C drive of a VM

This was tested on Windows 2003 R2

Here's a VM that need its C drive expanded:

Edit VM Setting and set the new VMDK size (this can be done online).

Rescan disks, and you'll see extra free space at the end of the partition.

Now boot into Parted Magic.
Select the first partition, click Resize and set the new size.

You will have to select Apply to actually do the resize.

Reboot into Windows.
CHKDSK will run automatically and finally you will have your expanded C drive.

Live Linux P2V

Here's how to clone a live, powered-on machine to a VMWare VM.

The test case scenario is a conversion from an HP BL460 to a Version 4 ESX VM, with LSI Logic parallel virtual SCSI controller and one e1000 vnic.
Linux version is Red Hat 5.5 x64
UPDATE: I've made some encouraging experiments with 6.2 also

Important: this method does not guarantee data consistency, as the disk will be copied while active, but it's usually good enought to bring up a bootable copy of the source machine where you can later restore a consistent backup.
On the other hand, this procedure gives way less downtime than a cold clone, and you can test the cloning process without halting the source machine.

Onward I will call the source machine "P" and the destination machine "V".

Take note of the P disk configuration

# fdisk -l /dev/cciss/c0d0

Disk /dev/cciss/c0d0: 146.7 GB, 146778685440 bytes
255 heads, 32 sectors/track, 35132 cylinders
Units = cylinders of 8160 * 512 = 4177920 bytes

           Device Boot      Start         End      Blocks   Id  System
/dev/cciss/c0d0p1               1          25      101984   83  Linux
/dev/cciss/c0d0p2              26       23961    97658880   8e  Linux LVM

This is an HP machine with a SmartArray controller.

Create a new VM, called V, with a disk at least as big a the P one: in this case I've created a 147GB vmdk.
Boot the V VM into rescue mode.

boot: linux rescue

Start the network interface and assign an IP enabled for SSH with the P machine.
Skip searching for existing installations.
All the following commands are to be executed on the V machine.
Now dump the P hard drive to the V vmdk:

# ssh root@P "cat /dev/cciss/c0d0 | gzip -1 | cat" | gzip -d >/dev/sda

note that this compresses the data over the wire, as this usually leads to faster transfers, even more if the disk has many empty sectors.

After completion, an fdisk should display the same partition table of the P machine on the V machine

# fdisk -l /dev/sda

Disk /dev/sda: 157.8 GB, 157840048128 bytes
255 heads, 32 sectors/track, 37779 cylinders
Units = cylinders of 8160 * 512 = 4177920 bytes

           Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          25      101984   83  Linux
/dev/sda2              26       23961    97658880   8e  Linux LVM


Reboot V into rescue mode again

boot: linux rescue

This time choose to search for existing installations, as it should find the new disk content.

# chroot /mnt/sysimage

Make sure the LVM has seen the new disk

# pvscan

PV /dev/sda2       VG VolGroup00   lvm2 [93.13 GB / 59.13 GB free]

If you don't see the PV, check /etc/lvm/lvm.conf for any filter other than the default one.

Modify the boot modules list taking as an example those of another identically configured VM.
This is the modprobe.conf of a tipical ESX VM.

# vi /etc/modprobe.conf

alias eth0 e1000
alias scsi_hostadapter mptbase
alias scsi_hostadapter1 mptspi
alias scsi_hostadapter2 ata_piix

Rebuild the initial ram disk, as referenced in the grub.conf

# mkinitrd -v -f /boot/initrd-2.6.18-194.el5.img 2.6.18-194.el5

Reinstall grub, just to be safe (it should be already there from the disk cloning process).

# grub-install /dev/sda

Now note that, after reboot, the V machine will start with the same IP address of the original P one.
If it doesn't suit your need, change the IP address in /etc/sysconfig/network-scripts/ifcfg-eth0
Now you can reboot the V machine.

If the M machine was at runlevel 5, the V machine may start with corrupted graphics because of the different video driver of the VM.
Switch to runlevel 3 and reconfigure X with

# system-config-display --reconfig

Now you can proceed with vmware-tools installation as usual on VMs.

How to unlock a VMDK file

while trying to power on an old VM i've got the following error message:

Unable to access file <unspecified filename> since it is locked

to investigate the issue i've looked and the vmware.log in the VM home folder and i've found the following error


Sep 07 10:18:24.260: vmx| Msg_Post: Error
Sep 07 10:18:24.260: vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/49b105df-0d080910-d97a-000423d87441/server/server.vmdk' or one of the snapshot disks it depends on.
Sep 07 10:18:24.260: vmx| [msg.disk.configureDiskError] Reason: Failed to lock the file.----------------------------------------

it seemed that the vmdk file was locked, but i was pretty sure that no other VM was using it and no snapshots were active.

after a little googling i've found the following thread that shows how to identify who is locking a vmdk file.

unfortunately, on my host it gave the following output:

[root@esx server]# vmkfstools -D server-flat.vmdk

[root@esx server]# tail /var/log/vmkernel

Sep  7 17:48:42 esx vmkernel: 36:03:13:22.953 cpu4:1041)FS3: 130: <START server-flat.vmdk>

Sep  7 17:48:42 esx vmkernel: 36:03:13:22.953 cpu4:1041)Lock [type 10c00001 offset 17129472 v 99, hb offset 3367936

Sep  7 17:48:42 esx vmkernel: gen 150, mode 0, owner 00000000-00000000-0000-000000000000 mtime 1213998]

Sep  7 17:48:42 esx vmkernel: 36:03:13:22.953 cpu4:1041)Addr <4, 21, 20>, gen 21, links 1, type reg, flags 0x0, uid 0, gid 0, mode 100600

Sep  7 17:48:42 esx vmkernel: 36:03:13:22.953 cpu4:1041)len 13958643712, nb 6656 tbz 0, zla 3, bs 2097152

Sep  7 17:48:42 esx vmkernel: 36:03:13:22.953 cpu4:1041)FS3: 132: <END server-flat.vmdk>

owner 00000000-00000000-0000-000000000000 was of no use to identify the culprit.

that same datastore was visible to several others esx hosts, so i tried the same command from another host and i had more luck: that time a full UUID was returned as the owner.

i've followed the original thread instructions to identify the esx host that was owning the lock and i've restarted it after migrating off all the VMs.

the reboot removed the lock and i was able to turn on again the old VM.

lesson learned: if at first you don't succeed, try on another host

Small & Quick VM to test network connectivity

It's quite some time I've been searching for a VM that allows to troubleshoot networking problems on ESX machines.
I've used DSL and his child Tinycore. They are good, they even offer a GUI that may be wasteful for simple network troubleshooting, but both lack tcpdump (at least in the downloadable ISO).
Now I've found something just right for the job.
PLD RescueCD: it's reasonably small (89MB ISO), runs entirely from RAM (as low as 40MB), it has tcpdump on board, nc, nmap and a load of other tools.
As a plus, it has no GUI :-)

I've used it to make sure the correct VLAN was mapped on a ESX machine:
Create a custom VM with 1 cpu, 64MB ram, no disk, 1 ethernet adapter bound to the port group you're investigating, and map the virtual CD to the PLD ISO file.
Boot up the VM and you'll be dropped to the root prompt.

#tcpdump

Even if ESX does not allow promiscuous mode on VM adapters, you'll still be able to see ARP requests from other machines that are on the same VLAN as your own VM.
That should be enough to understand if you've pinched the right VLAN.
Someone may say that ESX itself shows a network hint next to each adapter, but I've seen that such hint is not always reliable or at least is not real time, so a newly attached VLAN may not be immediately visible.

The case of the missing gateway

I've never quite understood the purpose of the NETWORK entry in the ifcfg file of the redhat-based linux distributions.
It can be derived from ip and mask, and if you leave it out altogether, the network works fine as well.
But if you decide to use it, be sure to write it correctly.
The other day I found an ESX machine (that's using redhat as the service console) that loses its default gateway after each reboot.
The gateway was correctly specified in the /etc/sysconfig/network file. After some fiddling I've noticed that the NETWORK clause in /etc/sysconfig/network-scripts/ifcfg-vswif0 was totally wrong. It's not even possible to set it with the vsphere client so I think someone mistyped it while manually configuring the ESX.
Once corrected, a

#service network restart

brought back the ESX online with the vcenter server.

Comparing Mhz from 2005 to 2011

i know i may asking for the holy grail here, but i'd like to share some thoughts on that.

a couple of months ago i was planning an hw refresh for a farm built way back in 2005, based on Xeon MP 3.0 Ghz processors, with newer blades with some X5660 processors.
looking at the vcenter statistics i can see the actual farm is using something like 50Ghz as an average, and has a physical to virtual cpu ratio of about 1.5.
by a rule of thumb, i may say that on newer processors, doubling the p2v cpu ratio won't cause any problem, even if i don't have any evidence to prove it.
speaking of Mhz instead, i'm trying to find out how a 2011 core compares to a 2005 one.
even if X5660 cores are running at 2.8Ghz, i don't think they're "slower" than Xeon MP ones.
i've fiddled a litte with the published spec.org CINT2006 benchmark results
here is a sample:

 xeon 3.8Ghz        11
e5345 2.33Ghz       16

e7330 2.4Ghz        17

e7458 2.4Ghz        20
x5460 3.16Ghz       27

x5560 2.8Ghz        35

so, it seems that the X5560 is at least 3 times faster than the old xeon.

after some weeks, i've stumbled upon this blog post that seems to come to the same conclusion
http://it20.info/2011/06/the-cloud-and-the-sunset-of-the-ghz-based-cpu-metric/
MIPS have served well for the mainframe world, why shouldn't they work for x86 also?

Linux V2P

Virtualization may be the best thing since the microprocessor itself, but sometimes you may have to backtrack your own footsteps.
Here are the steps to convert a Red Hat 5.5 linux vm running on an ESX to a physical box:

Using a disk imaging tool, dump the vmdk of the vm to the internal disk of the target machine.Boot the physical box with the Red Hat CD in rescue mode
Do not mount any partitions, but select "continue" to jump into the shell.

Mount the restored partition:

#chroot /mnt/sysimage

edit the modules list and put every driver the physical machine needs.
you can see which drivers are needed by looking at modprobe.conf from another machine that has the same hw as the one you are V2P-ing.
if you don't have another machine, you can use the same target machine: just before imaging it with the source vmdk, install it from scratch with the Red Hat CD and take note of the resulting modprobe.conf

#vi /etc/modprobe.conf

here is the one i used as a reference (an HP machine)

options bnx2x disable_tpa=1
alias eth0 bnx2
alias eth1 bnx2
alias eth2 tg3
alias eth3 tg3
alias scsi_hostadapter cciss
alias scsi_hostadapter1 lpfc
alias scsi_hostadapter2 usb-storage

then you will have to rebuild the initrd image to include the needed drivers.
note that kernel version number must match the one used on your vm.

/sbin/mkinitrd -v -f /boot/initrd-2.6.18-164.el5.img 2.6.18-164.el5

and reboot.
at boot time you will probably get some errors on X initialization because the graphic card has changed: just answer yes at the reconfigure option and select 800x600 as default resolution.

Update: if you are looking for P2V instead, look at this post.