invurted.com » Tutorials

[TUTORIAL] Failure to Enter Maintenance Mode

Adam — Fri, 03 Apr 2009 02:34:37 +0000

Greetings, faithful reader,

This one is as much a reminder for me as it is one to you!

When you’re entering Maintenance Mode, the Vmware ESX server will always try to Vmotion all of it’s hosted virtual machines away. Problems can occur when hosts fail to migrate when entering Maintenace Mode. The server will stop at two percent and then time out. Regrettably, there is no notification that an individual Virtual Machine’s Vmotion event fails.

However, there is another cause as well: if your cluster is both HA (High Availability) and DRS (Distributed Resource Scheduling) enabled, when you put an ESX server into Maintenance mode, DRS will generate a five star recommendation. Manual mode, however, requires user interaction and you have to tell the ESX cluster to initiate the Vmotion events. Once I accept the recommendations, Vmotion will start and the target ESX server will be placed in Maintenance Mode.

There is one other case where it will fail. This one falls into the category of “undocumented system feature”. The summary is:

From: http://kb.vmware.com/selfservice/viewContent.do?externalId=1007156&sliceId=1

An ESX host fails to enter maintenance mode in a VMware High Availability (HA) or DRS cluster

Hosts fail to migrate when attempting to enter maintenance mode

The progress indicator remains at 2% indefinitely

Trying to remediate a host and getting a time out error when trying to enter the maintenance mode

Cause:
This is normal behavior for a VMware HA/DRS cluster that is using strict admission control.

Disabling strict admission control (allowing virtual machines to power on even if they violate constraints) should allow a host to enter maintenance mode in this situation but a bug was discovered whereby it did not.
Resolution:

For a permanent solution, upgrade to VirtualCenter 2.5 Update 3.

To workaround the issue, temporarily disable VMware HA in the cluster settings. You will then be able to put the ESX Server host into Maintence mode and do the work required. You can then re-enable HA on your cluster.

[TUTORIAL] Boot ESX 3i from USB key

Adam — Wed, 11 Mar 2009 05:33:02 +0000

This one is getting filed under “Try it really soon!” However, randomly answering an email, I came across this document on http://www.vminfo.nl detailing the process for making a bootable USB key with ESX 3i installed.

Certainly, the document goes into a lot more detailed instructions, but here’s a bit of a summary:

Using an archiving program eg. IZArc, extract the contents of the Vmware ESX 3i ISO image. Given it’s an ISO, you could also mount it using DaemonTools or it’s like.
Once the files are extracted, find the INSTALL.TGZ file and open it.
Navigate to the \usr\lib\vmware\installer directory.
Open the Vmware-VMvisor-big-3.5.0_Update_2-11072.i386.dd.bz2 file. This may take a while, so patience, young padawan! Then extract the un-compressed .dd file
Restore the boot image to the USB thumb drive. Effectively, you’re restoring a “virtual hard drive” (in this case the .dd file extracted above.
Now that’s done, it’s time to test! Boot the sucker and play with ESX 3i!

The only real “gotcha” in this one is pretty obvious: make sure you have a motherboard that can boot from USB. Thankfully, these days most new motherboards have this ability. If in doubt check your motherboard manual or the manufacturer’s website.

Regrettably, I don’t have a font big enough OR bold enough to stress that this is UNSUPPORTED IN A PRODUCTION ENVIRONMENT!! Try it at your own risk!

ESX 3i on USB

[TUTORIAL] Force Virtual Machine to shutdown

Adam — Fri, 12 Dec 2008 03:38:52 +0000

Occassionally, when you try to shut down a virtual machine the progress bar will stop meaning that the process will never complete in the background. The following steps will allow you to force the shutdown of a running virtual machine.

Logon to the ESX host where the virtual machine is running and become root.
vmware-cmd -l to list all the registered virtual machines.
vmware-cmd /path/copied/from/vmware-cmd getstate to get state of the target virtual machine.
If the state requires an answer:
- vmware-cmd /path/copied/from/vmware-cmd answer
If no answer is needed:
- vmware-cmd /path/copied/from/vmware-cmd stop trysoft
If trysoft does not work use vmware-cmd /path/copied/from/vmware-cmd stop hard
If the vmware-cmd does not work, the next step is to kill the master user world id.
cat /proc/vmware/vm/*/names |grep vmname replace vmname with the name of the virtual machine that has hung. From this list, get the VMID
less /proc/vmware/vm/vmid value/cpu/status where vmid value is the number from above.
Scroll over to the right until you find the group field that shows vm.####. The #### numbers are the master user world id.
/usr/lib/vmware/bin/vmkload_app -k 9 #### where #### is the master user world id. If the command is successful you will get a WARNING message that a signal 9 is being sent.
If vmkload_app does not help the next thing to try is to crash the virtual machine with the vm-support -X command.
vm-support -x to get the vmid.
From a directory that has ample space vm-support -X #### where #### is the vmid.
Answer all the questions with the default answers. The entire process takes about 10 minutes and creates an archive log that can be submitted to support. It will also crash the vm

iSCSI security

Adam — Thu, 11 Dec 2008 00:48:45 +0000

iSCSI, as a storage option, has less performance than fibre channel ie. it is limited to the speed of the HBAs or the network card being used, however it is far cheaper and I can leaverage my existing infrastructure to put in an iSCSI SAN. The problem becomes, in a TCP/IP network, how do I protect the data moving between the SAN and the hosts?
To start with, what is iSCSI? Functionally, there is no difference between Fibre Channel, iSCSI and a local SCSI controller. All three send SCSI commands to hard disks. Both Fibre Channel and iSCSI transmit over fiber and TCP/IP networks, respectively; making the network storage appear as “local” storage for the iSCSI client (or initiator).
As I mentioned, iSCSI uses TCP/IP networks to transport the SCSI commands from the initiator to the target (the iSCSI SAN). iSCSI uses the default TCP port 3260 and provides no native security on communication between the initiator and the target.
Here’s where the challenges begin.
To start with, if my iSCSI traffic is shared on a production network there are two problems. The most dangerous is network sniffers for obvious reasons.
The other immediate problem is network contention. If my iSCSI SAN shares the same network as all my production network there is a big problem with performance as my normal traffic will be competing for access with iSCSI.
The solution to both problems is network isolation. This isn’t so much best practice as it is mandatory. Wether the iSCSI network is physically isolated to it’s own infrastructure or using IEEE 802.1Q (VLAN tagging), the iSCSI network should now be invisible and/or inaccessible to the production network. The only notable exception becomes the requirement for servers to access the iSCSI implementation. Easily fixed by the addition of network adapters to the servers. Even if I need to populate each server with additional NICs, this should still be cheaper than the most simple Fibre Channel implementation.
Okay, now we have isolation to provide security but what about securing the communication itself.
The obvious solution is IPSec. The not so obvious problem is the overhead to IPSec. For a storage network, it doesn’t seem optimal to have to encrypt and decrypt each packet as it is transmitted and received. Certainly, as of the writing of this article, there is no support for IPSec in ESX3.5 specifically.
So the only real option left (in ESX particular) is authentication to verify my identity. ESX supports CHAP authentication only for iSCSI initiators and targets.
CHAP (Challenge Handshake Authentication Protocol) is used to verify the identity of clients in a Point-to-point network. Your identity is verifed by using a three way handshake. After the initial link CHAP authentication is done at random intervals during the entire conversation. The one weakness is the reliance on a shared secret ie. the client’s password. ESX does not support per target credentials either. So one CHAP username and password sent to all iSCSI targets.
The CHAP process goes something like the following:

After the completion of the link establishment phase, the authenticator sends a “challenge” message to the peer.
The peer responds with a value calculated using a one-way hash function, such as an MD5 checksum hash.
The authenticator checks the response against its own calculation of the expected hash value. If the values match, the authenticator acknowledges the authentication; otherwise it should terminate the connection.
At random intervals the authenticator sends a new challenge to the peer and repeats steps 1 through 3.

An example of CHAP authentication can be seen between two Cisco 3640 routers. The network is a fairly simple on and looks like this:

After enabling the serial interfaces between the two routers, I enable and configure PPP encapsulation. On the second router (R2) , I enable CHAP authentication and view debugging information on the first router (R1)

*Mar 1 00:25:39.971: Se1/0 PPP: Using default call direction *Mar 1 00:25:39.975: Se1/0 PPP: Treating connection as a dedicated line *Mar 1 00:25:39.975: Se1/0 PPP: Session handle[26000001] Session id[0] *Mar 1 00:25:39.975: Se1/0 PPP: Authorization required *Mar 1 00:25:40.255: Se1/0 PPP: No authorization without authentication *Mar 1 00:25:40.259: Se1/0 CHAP: I CHALLENGE id 1 len 23 from "r2" *Mar 1 00:25:40.299: Se1/0 CHAP: Unable to authenticate for peer
R2 is waiting for authentication from R1 and is unable to get authentication information. Now I configure CHAP authentication on r1 and observe the following:
*Mar 1 00:26:26.867: Se1/0 LCP: Received AAA AUTHOR Response PASS *Mar 1 00:26:26.867: Se1/0 IPCP: Received AAA AUTHOR Response PASS *Mar 1 00:26:26.871: Se1/0 CHAP: O SUCCESS id 12 len 4 *Mar 1 00:26:27.055: Se1/0 CHAP: I SUCCESS id 15 len 4 *Mar 1 00:26:27.059: Se1/0 PPP: Sent CDPCP AUTHOR Request *Mar 1 00:26:27.063: Se1/0 PPP: Sent IPCP AUTHOR Request *Mar 1 00:26:27.071: Se1/0 CDPCP: Received AAA AUTHOR Response PASS *Mar 1 00:26:28.055: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0, changed state to up
The shared secret is exchanged between the two and the Serial interface is changed to an up state.

[PRACTICE] Fixed length subnet masking #2

Adam — Wed, 10 Dec 2008 04:53:04 +0000

Following on in the series, here is the second subnet masking example.

You have been given the network range 192.168.128.0/24. Your task is to break it down into six networks. The questions you have to answer are, how many hosts on each network, and what are the network IDs and broadcast IDs for each of the networks.

I will post the answer on Friday 12th December, 2008 at 3:30pm EDST.

Sorry for the lateness of the solution. I have been as sick as a dog for the last few days, hence no update.

Last time we talked breaking networks up based on host requirements. This time we’re working with the network portion of the IP address.

When we were working with hosts, we worked from the thirty-second bit. Or, from right to left. This time we’re working from left to right to break up an IP range.

Let’s look at the IP address in binary first. Because it’s a twenty-four bit subnet mask, this tells us that it’s a Class C network and that the first three octets are locked and can’t be changed. Therefore, we will be breaking up the fourth octet.
192.168.128.00000000 255.255.255.00000000

Similarly to subnetting for hosts, subnetting for networks is achieved by modifying the subnet mask. So rather than working from the last bit leftwards, let’s start with the first bit I am allowed to modify and work to the right. In our example, the twenty-fifth bit (the first bit of the fourth octet).

For each bit I change, I get 2^bits networks. Unlike hosts, I don’t lose the first IP address to the network ID or the last IP address to the broadcast. Therefore, if I use one bit I get two usable networks. If we extend this to cover our example, two bits gives me four networks (still not enough!). Can I get exactly six networks? Because I’m working with binary values (powers of two), it’s not possible to get exactly six networks. If I use three bits, I get eight networks. More than the required six.

First thing’s first: the subnet mask. The original Class C has been subnetted. Now I am using the first three bits of the fourth octet as the network portion. It did look like this: 255.255.255.00000000. Now, after subnetting, it looks like this: 255.255.255.11100000 or, (after converting back to decimal) 255.255.255.224.

My networks look something like this (the spaces represent the division between my new network ID and new host ID):
192.168.128.000 00000 192.168.128.001 00000 192.168.128.010 00000 192.168.128.011 00000 192.168.128.100 00000 192.168.128.101 00000 192.168.128.110 00000 192.168.128.111 00000

How many hosts on each network? Seeing that we are using three bits for the network, the remaining five bits are the host portion ie. (2^5)-2 hosts or 32-2 hosts. A total of 30 hosts per network. Going back to the last tutorial, the host portion can’t be all zeroes, nor can it be all ones.

Therefore, my network ranges look like this:
192.168.128.000 00000 192.168.128.000 11111 192.168.128.001 00000 192.168.128.001 11111 192.168.128.010 00000 192.168.128.010 11111 192.168.128.011 00000 192.168.128.011 11111 192.168.128.100 00000 192.168.128.100 11111 192.168.128.101 00000 192.168.128.101 11111 192.168.128.110 00000 192.168.128.110 11111 192.168.128.111 00000 192.168.128.111 11111

Any questions or problems, please email me contactme@invurted.com or my work email address.

[PRACTICE] Fixed length subnet masking #1

Adam — Mon, 08 Dec 2008 01:37:00 +0000

After the other week’s CCENT course, I made a promise to start emailing subnetting examples out. Rather than writing and re-writing the same thing time and time again, I figured I’d put them up here instead.

So, fixed length subnet masking example #1 is this:

You have been given the network range 192.168.128.0/22. Your first task is to break this network up into blocks of one hundred IP addresses. How many blocks of IP addresses will you get (how many networks will you have)? What are the network IDs and what are the broadcast IP addresses for each of the network ranges?

I will post the answer on Tuesday 9th December, 2008 at 10:30am EDST.

Okay. The question specifically asks for hosts. Therefore, we are working from the thirty second bit back to the start to get hosts.

The original subnet mask is a /22 (255.255.252.0). This tells me that the first twenty two bits are “locked away” leaving me the last ten bits to play with. Straight away we can discern at least one piece of information about my network: there are 2^10 total hosts in the network ie. 2^#bits that aren’t locked away, or one thousand and twenty four total hosts.

On to the good stuff. If we look at the IP address in it’s binary format we have to do a decimal to binary conversion on the third and fourth octets. By looking at the subnet mask this tells me it’s not a full block of eight bits in the third octet and the fourth is all zeroes.

Which looks something like this:
192.168.10000000.00000000 255.255.11111100.00000000
Notice twenty two contiguous ones and ten contiguous zeroes.

As previously mentioned, I want hosts. So, I am working from the last bit leftwards. Each zero represents two hosts (1 and 0). Every time I go up a column, I am adding a power of two to the total number of hosts. If I use one column (the thirty-second bit), I have 2^1 hosts, two columns gives me 2^2 hosts etc. Can I get exactly 100 hosts for the network? The answer is no. If I use the last six bits I have sixty-four hosts (2^6), well below the required one hundred hosts! Let’s use the last seven bits. This gives me 2^7 hosts. One hundred and twenty eight is well above the one hundred required, but is the nearest value over one hundred I get given I have to count in powers of two.

My new subnet mask now looks something like this:
192.168.100000 00.0 0000000 255.255.111111 11.1 0000000
The spaces show the modifications that I have made to the network range given the requirements of the scenario.

The three bits between the spaces are telling me what networks I am now defining. Three bits, straight away, tells me I have 2^3 or eight networks in total. The network part of the IP address has to be unique. The eight unique combinations of the network ID are: 192.168.100000 00.0 0000000 (192.168.128.0) 192.168.100000 00.1 0000000 (192.168.128.128) 192.168.100000 01.0 0000000 (192.168.129.0) 192.168.100000 01.1 0000000 (192.168.129.128) 192.168.100000 10.0 0000000 (192.168.130.0) 192.168.100000 10.1 0000000 (192.168.130.128) 192.168.100000 11.0 0000000 (192.168.131.0) 192.168.100000 11.1 0000000 (192.168.131.128)

The rules of TCP/IP state that the host ID can’t be all zeroes and it can’t be all ones. The all zeroes address is the network ID and the all ones address is the broadcast IP address for that network.

In our example, the last six bits are the host ID. Therefore, the network IDs are up above and the broadcast IP addresses are:192.168.100000 00.0 1111111 (192.168.128.127) 192.168.100000 00.1 1111111 (192.168.128.255) 192.168.100000 01.0 1111111 (192.168.129.127) 192.168.100000 01.1 1111111 (192.168.129.255) 192.168.100000 10.0 1111111 (192.168.130.127) 192.168.100000 10.1 1111111 (192.168.130.255) 192.168.100000 11.0 1111111 (192.168.131.127) 192.168.100000 11.1 1111111 (192.168.131.255)

Any questions or problems, please email me contactme@invurted.com or my work email address.

Stay tuned because I’ll post another example later this week.

[TUTORIAL] Releasing Virtual Machine Files

Adam — Tue, 28 Oct 2008 03:42:55 +0000

Sometimes a file or set of files in a VMFS become locked and any attempts to edit them or delete will give a device or resource busy error, even though the virtual machine associated with the files is not running. If the virtual machine is running then you would need to stop the virtual machine to manipulate the files. If you know that the virtual machine is stopped then you need to find the ESX server that has the files locked and then stop the process that is locking the file(s).

Logon to the ESX host where the virtual machine was last known to be running.
vmkfstools -D /vmfs/volumes/path/to/file to dump information on the file into /var/log/vmkernel
less /var/log/vmkernel and scroll to the bottom, you will see output like below:
Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 130: < START vmware-16.log > Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Lock [type 10c00001 offset 30439424 v 21, hb offset 4154368Nov 29 15:49:17 vm22 vmkernel: gen 66493, mode 1, owner 46c60a7c-94813bcf-4273-0017a44c7727 mtime 8781867] Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Addr <4, 588, 7>, gen 20, links 1, type reg, flags 0x0, uid 0, gid 0, mode 644 Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)len 23973, nb 1 tbz 0, zla 2, bs 65536 Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 132: < END vmware-16.log >
The owner of the lock is on the third line, the last part is all you need, in this case 0017a44c7727
esxcfg-info | grep -i ‘system uuid’ | awk -F ‘-’ ‘{print $NF}’ will display the system uuid of the esx server. You need to run the esxcfg-info command on each esx server in the cluster to discover the owner.
When you find the ESX server that matches the uuid owner, logon to that ESX server and run the command: ps -elf|grep vmname where vmname is the problem virtual machine. Example output below:
4 S root 7570 1 0 65 -10 - 435 schedu Nov27 ? 00:00:02 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user/pool2 -@ pipe=/tmp/vmhsdaemon-0/vmxf7fb85ef5d8b3522;vm=f7fb85ef5d8b3522 /vmfs/volumes/470e25b6-37016b37-a2b3-001b78bedd4c/iu-lsps-vstest/iu-lsps-vstest.vmx0
Since there is a process running, pid 7570 in the example, you need to kill it by following steps 5-12 on Stopping a Virtual Machine.
Once the kill is complete the files should be released.

[TUTORIAL] Stopping a Virtual Machine

Adam — Sun, 26 Oct 2008 03:23:11 +0000

VMware has added a new command in VI3 to help when a user has a VM that has become unresponsive. Below are the progressives steps to go through to get the VM cleanly powered off. Whatever you do, DO NOT kill the pid for the VM from the Service Console, unless you have tried absolutely everything. Killing the pid from the Service Console may prevent the VM from restarting.

The following steps assume that the usual graceful shutdowns do not work from within the VM operating system or the Virtual Infrastructure Client. Commands are bold underscore type, with descriptions of path names and some example output is included.

Logon to the ESX host where the VM is running and become root.
vmware-cmd -l to list all the registered VMs.
vmware-cmd /path/copied/from/vmware-cmd getstate to get state of vm.
If the state requires an answer: vmware-cmd /path/copied/from/vmware-cmd answer
If no answer is needed: vmware-cmd /path/copied/from/vmware-cmd stop trysoft
If “trysoft” does not work: vmware-cmd /path/copied/from/vmware-cmd stop hard
If the vmware-cmd does not help next up is to kill the master user world id
cat /proc/vmware/vm/*/names |grep vmname where vmname is the vm that is hung and find the value for vmid
less /proc/vmware/vm/vmid value/cpu/status where vmid value is the number from above
Scroll over to the right until you find the group field that shows vm.#### where the #### numbers after vm. will be the master user world id
/usr/lib/vmware/bin/vmkload_app -k 9 #### where #### is the master user world id. If successful you will get a WARNING message that a signal 9 is being sent
If vmkload_app does not help next up is to crash the vm with the vm-support -X command
vm-support -x to get the vmid
From a directory that has ample space vm-support -X #### where #### is the vmid
Answer all the questions with the default answers. The entire process takes about 10 minutes and creates an archive log that can be submitted to support. It will also crash the vm.

[TUTORIAL] Show hidden devices after P2V

Adam — Fri, 10 Oct 2008 22:41:04 +0000

After performing a P2V always remove the hidden physical hardware from the OS. This is particularly important for network cards that have the original IP address(es) that you want to assisgn to the new VM.

1. Click Start | Run | cmd

2. At a command prompt, type the following command , and then press ENTER:

set devmgr_show_nonpresent_devices=1

3. Type the following command in the same command prompt window, and then press ENTER:

start devmgmt.msc

4. Click Show hidden devices on the View menu in Device Managers before you can see devices that are not connected to the computer.

[TUTORIAL] Migrating Virtual Machines without Vmotion

Adam — Thu, 02 Oct 2008 06:50:03 +0000

Guys are hard at work with the labs at the moment here in sunny Hobart, and so I begin trawling the blogs (as is my want). I discovered this brilliant little gem from Mike DiPetrill’s blog.

This post demonstates the awesome power of the Vmware API and the ease of use we get from the VI Toolkit for Windows! The script will move a VM from one ESX host to another, without making use of VMware VMotion. The drawback is that the VM becomes unresponsive for a period of time (usually a few seconds). This is roughly functionally equivalent to Microsoft’s Quick Migration feature, except that this was done in 130 lines of script.