Frantic calls to and from on Saturday night lead myself and a friend to the question, “What the hell is up with iSCSI?”

It seems that the initiator works only with a service console. In fact, the observed results showed that iSCSI was being run with the Service Console’s IP address DESPITE have a vmkernel port configured in the same networks as the iSCSI target.

Well, this is not entirely true. Vmware needs a Service Console to do the initial CHAP authentication and establishment and only then does it switch over to the configured vmkernel interface.

Why even bother with an iSCSI “interface” (vmkernel port0 if you’re going to initiate with the Service Console? In fact, more to the point, why does it split the authentication and actual iSCSI traffic across them.

Based on various forum posts, it sounds like iSCSI was a last minute throw in by VMware, and just not done completely right.

Hence, Vmware Vista.

REF: my mobile phone log with the lingfish and http://communities.vmware.com/thread/209361

Specifically, in the environment I am in at the moment, there is a requirement for the operating system to be deployed to one LUN and the any application partitons to be deployed to another LUN.

Templates only allow me to deploy the machine’s virtual hard disk to one LUN and not split across two. Hence the problem.

There is no quick fix that I could find; so it was off the scripting land I went.

First things first. The customer required the deployment of twenty seven virtual machines all based off one. The process is prety simple; open my Virtual Infrastructure Client, point it at a Virtual Center (ESX servers by themselves don’t have template functionality), and convert the baseline virtual machine to a template.

I could have chosen to deploy twenty seven virtual machines manually, but, well, I’m lazy. So, the lazy man’s solution? Scripting!

The process was as follows:

1) Remove the 40GB hard drive (cloning and templating doesn’t allow hard drives to be placed in multiple datastores eg. osLUN and appLUN)
2) Create a template of the virtual machine.
3) Install powershell and the Vmware VI Toolkit giving us the powershell for Vmware functionality.
4) Save the following to a .ps1 (Powershell script file):

$array = “Machine1″,”Machine2″,”Machine3″ (and so forth)
$TARGET = “FQDN_OF_TARGET_ESX_SERVER”

Connect-viserver FQDN_OF_VIRTUAL_CENTER

foreach ($vm in $array)
{
$vm=New-VM -Name $vm -Template BASELINE_TEMPLATE -Host $TARGET -Datastore osLUN
}

The above script will connect to the VI Center (it will prompt for username and password) and then create a new machine called “Machine1″ etc. for each machine in the list. It will place the virtual machine on the esx server in $TARGET and build it from the template called BASELINE_TEMPLATE.

Now comes the hard part. There are 27 42GB drives that need to be created for the virtual machines. There is no powershell that will allow this to happen. Therefore, a script is required.

In appLUN is a vmdk called BASELINE_TEMPLATE_VMDK and this is the virtual hard drive (pre partitioned and formatted) that will be used for the virtual machines.

for DESTINATION in machine1 machine2 machine3 machine4 ETC.
do
# Copy
echo “Copying $SOURCE to $DESTINATION”
mkdir $DESTINATION
cp -a $SOURCE/* $DESTINATION/
# Rename vmdk files
cd $DESTINATION
for file in $(ls -1 *.vmdk)
do
echo “Renaming $SOURCE vmdks to $DESTINATION”
DEST=$(echo $file | sed s/$SOURCE/$DESTINATION/g)
mv $file $DEST
done
# Edit .vmdk
for file in $(ls -1 *.vmdk | grep -v flat)
do
echo “Replacing references to $SOURCE in .vmdk file $file”
sed -i s/$SOURCE/$DESTINATION/g $file
done
cd ..
done
echo “Finished”

This will copy the BASELINE_TEMPLATE_VMDK virtual hard drive one time for each virtual machine and make sure it is useable for that machine. It will also go through the .vmdk file and change the reference from BASELINE_TEMPLATE_VMDK to the destination virtual machine name.

Once it is run, we have to manually add the newly created vmdk to the corresponding virtual machine.

Then it is a simple matter of logging into each virtual machine and changing their host names and IP addresses.

There is still a fair bit of manual intervention involved, but at least the boring, tedious bits (deploying the templates and copying the hard drives) has been automated.

If you can offer any other scripts that can complete the picture, let me know!

1) How often ESX checks for path failure?
==> As soon as an I/O request to a path fails, ESX will initiate a path failover. If there is no I/O outstanding to a path. ESX will probe each physical path every 5 minutes, by default, to proactively detect path failure.

2) How long will ESX wait before trying a different storage path?
==> A different storage path is tried immediately.

3) What happens between the failure detection and the connection recovery?
==> I/O requests will be queued.

4) Under which circumstances will a host initiated target reset or LUN reset occur?
==> Resets are not typically initiated by ESX. Two exceptions to this rule are:
- if a path failure occurs while there is a SCSI-2 reservation outstanding on the failed path
- if the VM or userworld that initiated the I/O request sends a request to abort the outstanding I/O request.

5) Will ESX ever force a LUN trespass in the array?
==> Only in the case of an A/P array when there is no working path on the same SP as the failing path. For example, when using an EMC Clariion (which uses the “trespass” command).

Technically point two needs some work.

ESX SCSI layer has a latency between when the failure occurs and when it will try a different path. This is usually determined by when the device driver returns a failure ie. I/O errors. Supported fibre channel devices will do this inside of thirty seconds. The ESX SCSI layer then has thirty seconds to use another working path. On an active/passive array the activate/trespass/etc command can take a few seconds to complete.

Within a virtual machine, the registry ([HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Disk]
“TimeoutValue”=dword:000000be) can be configured to wait before issuing an abort for an outstanding I/O. If this key is enabled

ESX states that within this 60 seconds it will:
- detect that a path has failed
- select a new path
- activate the new path
- re-issue the cmd from the guest to the new path
- have the newly issued cmd complete successfully and return to the guest.

So, whilst vSphere is the new kid on the block, Vmware have also released an updated version to this genuinely brilliant utility!

What’s New
The VMware vCenter Converter Standalone 4.0.1 release is an update to Converter Standalone 4.0, and includes the following new features:

Support for vSphere 4.0 as source and destination targets:
• Support for configuring target disks as thin provisioned disks
• Support creation of IDE disks on vSphere 4.0
• Support for backup products to restore vSphere 4.0 virtual machines backed up using VCB
• Support for creation of virtual hardware version 7.0 virtual machines on vSphere 4.0. targets as well as migration of hardware version 7.0 virtual machines from Workstation and Server platforms to vSphere 4.0
• Support for importing OVF 1.0 single virtual machine images.
• Support for customization of Windows Server 2008 guests.

The twitter-sphere and the blog-verse is going mad with live reports from Vmware’s announcements.

Specifically, Duncan Epping’s blog has some interesting links: http://www.yellow-bricks.com/2009/04/21/vsphere-linkage/

Also, twitter search for vSphere, or check out Scott Lowe’s blog (http://blog.scottlowe.org/) and also Rick Scherer (http://vmwaretips.com/wp/).

And for an interesting response, Microsoft’s reaction to the announcement: http://darow.wordpress.com/2009/04/21/reactions-to-vmwares-vsphere-4/

It’s a shame I’m stranded here in Australia and can’t attend these events easily!

This one is as much a reminder for me as it is one to you!

When you’re entering Maintenance Mode, the Vmware ESX server will always try to Vmotion all of it’s hosted virtual machines away. Problems can occur when hosts fail to migrate when entering Maintenace Mode. The server will stop at two percent and then time out. Regrettably, there is no notification that an individual Virtual Machine’s Vmotion event fails.

However, there is another cause as well: if your cluster is both HA (High Availability) and DRS (Distributed Resource Scheduling) enabled, when you put an ESX server into Maintenance mode, DRS will generate a five star recommendation. Manual mode, however, requires user interaction and you have to tell the ESX cluster to initiate the Vmotion events. Once I accept the recommendations, Vmotion will start and the target ESX server will be placed in Maintenance Mode.

There is one other case where it will fail. This one falls into the category of “undocumented system feature”. The summary is:

From: http://kb.vmware.com/selfservice/viewContent.do?externalId=1007156&sliceId=1

• An ESX host fails to enter maintenance mode in a VMware High Availability (HA) or DRS cluster
• Hosts fail to migrate when attempting to enter maintenance mode
• The progress indicator remains at 2% indefinitely
• Trying to remediate a host and getting a time out error when trying to enter the maintenance mode

Cause:
This is normal behavior for a VMware HA/DRS cluster that is using strict admission control.

Disabling strict admission control (allowing virtual machines to power on even if they violate constraints) should allow a host to enter maintenance mode in this situation but a bug was discovered whereby it did not.
Resolution:

For a permanent solution, upgrade to VirtualCenter 2.5 Update 3.

To workaround the issue, temporarily disable VMware HA in the cluster settings. You will then be able to put the ESX Server host into Maintence mode and do the work required. You can then re-enable HA on your cluster.

Certainly, the document goes into a lot more detailed instructions, but here’s a bit of a summary:

1. Using an archiving program eg. IZArc, extract the contents of the Vmware ESX 3i ISO image. Given it’s an ISO, you could also mount it using DaemonTools or it’s like.
2. Once the files are extracted, find the INSTALL.TGZ file and open it.
3. Navigate to the \usr\lib\vmware\installer directory.
4. Open the Vmware-VMvisor-big-3.5.0_Update_2-11072.i386.dd.bz2 file. This may take a while, so patience, young padawan! Then extract the un-compressed .dd file
5. Restore the boot image to the USB thumb drive. Effectively, you’re restoring a “virtual hard drive” (in this case the .dd file extracted above.
6. Now that’s done, it’s time to test! Boot the sucker and play with ESX 3i!

The only real “gotcha” in this one is pretty obvious: make sure you have a motherboard that can boot from USB. Thankfully, these days most new motherboards have this ability. If in doubt check your motherboard manual or the manufacturer’s website.

Regrettably, I don’t have a font big enough OR bold enough to stress that this is UNSUPPORTED IN A PRODUCTION ENVIRONMENT!! Try it at your own risk!

ESX 3i on USB

But here’s another reason to be careful about applying updates without considering your environment.

There is a firewall on the service console to protect it from the network. To ensure it is relativley secure, the firewall only has opened ports for required services. For example, when I enable software iSCSI, ESX SHOULD open up port 3260 outbound.

Interestingly enough, I can SSH to an ESX server from an external host, but, by default, can’t SSH from an ESX server once I am connected. This is because tcp/22 is NOT enabled outbound by default.

In certain environments there are other considerations to take into account as well. For example, is ESX’s firewall EAL 4+ European Accredited or TESG Accredited

The answer is kind of interesting. ESX 3.0.2 has been EAL4+ so this would include all components, including the firewall. ESX 3.5 U2 is being reviewed at this point and there is no fixed date as to when it will be certified.

Welcome to Vi-Pedia, the Virtualization Open Wiki. The purpose for this wiki is to gather information on all virtualization technologies in one place. This is a community driven project and anyone can therfore update the wiki.

An interesting resource, to be certain. Could this mean the end of yours truly as the wiki blows him out of the water?

Only time will tell, gentle reader, only time will tell.
http://www.vi-pedia.com/

From time to time I am asked if there were any benefit to templating vs. cloning vs. storing a VM in OVF format. So I thought I’d use the resources at my disposal from my VDC kit and give it a go.

We all know that it can take longer to create a compact template than a normal template. However on the flip side, a compact template normally deploys more quickly. We also know that storing a template in compact format may possible save us room in storage depending on how much free space there is in the vmdk(s).

I took our normal class VM (2GB hard drive) and created a compact template and then exported it to OVF format virtual appliance. Then I compared the resulting sizes of each. OK, I was not going for “exact” numbers, just general indications of size (see attached graphics).

1. Original VM took slightly more than 2GB to store (as would the resulting normal template)
2. Compact template took slightly more than 1GB to store
3. OVF virtual appliance took slightly less than 500MB to store

This seems to indicate that there is a significant space savings to be had if you store your VMs as OVF virtual appliances instead of templates or clones. Now this does not take in to account the time it takes to create and deploy VMs from OFV vs. templates, but it did get me to thinking. I would like to try this again with a larger vmdk and with more actual data saved to the vmdk, but at the moment I have neither the time nor the resources to do so.

Original Virtual Machine
Compact Template
OVF

Aaron Thorpe, a fellow VCI said, It’s definitely something interesting to consider. One thing we discovered during a class discussion was that if you create a compact template, it converts the VMDK to a thin disk. However all new VMs created from this template will also use a thin disk. We were concerned that this may hurt disk performance of all future VMs deployed from the template, but we did not have time to test thoroughly.

I wonder if anyone has written a comparative whitepaper…

That said, I think that templates have more benefits than OVF if we are talking about VMware features rather than just saving space. Cloning still allows us to use the “Guest Customization Wizard”. I can’t do this to an OVF machine.