invurted.com

Virtualisation; I don’t get it

Adam — Mon, 01 Mar 2010 23:30:05 +0000

It seems that in the local industry (Melbourne, Australia) that businesses just don’t get virtualisation.

At the cessation of another job, I am beginning to wonder if this virtualisation thing will ever catch on. Or maybe I’m too far ahead of the curve and waiting for it to catch up.

Let’s start with the benefits;

no reliance on physical infrastructure
Rapid provisioning of both servers and desktops (VDI)
Fault tolerance and, if I want it, automated load balancing
Cost savings in the datacentre because I’m not taking up as much space with hardware that needs to be maintained and replaced as often

The disadvantages are far outweighed by the benefits. It does cost to put in a virtual infrastructure; primarily the storage. Fibre channel is still “king” in the storage stakes, however, advances in ethernet technologies and a bit of planning in the implementation phase mean that iSCSI is a viable competitor (http://invurted.com/iscsi-vs-fibre-channel-storage-performance/). Of course, there are training considerations. Thankfully, the virtualisation platform is largely “fire and forget” technology. So apart from some familiarity with my chosen platform, there is little reason to touch it post-implementation.

Maybe I’m missing the point somewhere …

iSCSI vs. Fibre Channel storage performance

Adam — Sun, 21 Feb 2010 23:14:28 +0000

It’s something that I have been discussing a lot with one of my customers.

At the moment, they are hesitant to invest in a potentially very expensive SAN solution involving fibre channel connections. Currently, there are eight vSphere servers configured with no centralised storage. Hence, all virtual machines are running on the local disk arrays. Therefore, there is no DRS, no HA and no Vmotion! We are protecting the virtual machines only with the local server’s ability to recover from physical disk failure, in this case, on board RAID 5 controllers.

My temporary solution was an open source iSCSI storage solution until a more permenant solution can be found.

However, the big question that I keep coming up against is, “What’s the performance hit I will take?”

It’s undeniable that the fibre channel should be faster, as the article below describes, this may not be that much of an issue.

If we look at the way a virtual machine (and a physical machine, indirectly) runs, the most important resources are RAM and CPU. Disk storage is (sort of) secondary. The exception to this is streaming media or backup servers. Typically, performance difference is minimal.

As long as you have a properly implemented iSCSI solution. I have look at the security considerations previously (http://invurted.com/tutorial-iscsi-security/) and performance will always be optimal if iSCSI is isolated to its own infrastructure. This can be achieved by physical infrastructure or VLANs to isolate the traffic.

In short, there should be very little to stop small to medium enterprises from adopting iSCSI solutions for shared storage. Its performance is comparable to fibre channel in most circumstances and the relative cost is less than most fibre channel solutions for a minimal performance hit.

What weighs more: one pound of bricks or one pound of feathers? Which is faster: 2 Gb FC or 1 Gb Ethernet? Hint: Both questions have the same answer.

The area of iSCSI performance and how it compares to Fibre Channel is often misunderstood. Both of these SAN interconnects are typically measured by bandwidth with “2 Gb” FC SANs dominating the market today and “1 Gb” Ethernet used for the majority of iSCSI SANs.

Which would you say is faster: a 2 Gb FC connection or a 1 Gb Ethernet connection? It’s a trick question — they are equally fast. They both transfer data at the speed of light. Bandwidth is not an issue of speed but size. Tthink about a four-lane highway versus a two-lane highway. If there are just a few automobiles traveling on either highway, drivers will be able to go the maximum speed. As more drivers travel on each road, the two-lane highway will experience a bottleneck before the four-lane highway does.

This is the same with FC and Ethernet. A 2 Gb FC interconnect has twice the bandwidth (double the number of lanes) of 1 Gb Ethernet. Bandwidth has an impact on performance when large requests are being processed. In this case, most of the work is spent transferring the data over the network making bandwidth the critical path. However, for smaller read and write requests, the storage system spends more time accessing data making the CPU, cache memory, bus speeds and hard drives more important to overall application performance.

Unless you have a bandwidth-intensive application (e.g., streaming media or backup data), the difference in performance will be minimal. Enterprise Strategy Group (ESG) Lab has tested storage systems that support iSCSI and FC and the performance difference is minimal — ranging between 5% and 15%.

In fact, an iSCSI storage system can actually outperform a FC-based product depending on more important factors than bandwidth, including the number of processors, host ports, cache memory and disk drives and how wide they can be striped.

The slowest component of the storage performance chain is the hard disk drive. It takes a hard disk drive much longer — sometimes several thousands-percent longer — to access data in a storage system than the electronic components like processors, bus and memory. The timeline for an I/O starts with a read/write command being sent to the hard drive from the application. This is followed by long, mechanical access times waiting for the drive to move the actuator, referred to as the seek process.

The seek process is by far the slowest part of storage performance. The actuator then has to spin to the data that’s been requested, which is another long mechanical process that creates latency. Next, the data is transferred from the drive to the CPU and a status handshake is performed to terminate the request. Access time associated with all disk drives, which includes seek plus latency, is responsible for the majority of the “wait time.”

Striping data
Traditional storage systems are typically limited in the number of drives across which they can stripe data. Many traditional storage systems can only stripe up to 16 drives, while more advanced products can stripe across hundreds of drives. Striping data across a large number of drives allows a system to leverage all the actuators, which work in parallel to make read/write functions a much more efficient process. Striping data across many drives increases performance and essentially eliminates the need for tuning performance and determining hot spots. Naturally, there is a cost associated with acquiring more hard drives, so a balance and consideration of price/performance is important.

In ESG Lab head-to-head testing, we configured a storage system using traditional striping methods and another one using wide striping. ESG Lab used the same workloads to compare the performance of the traditionally configured system and that of a system using a wide stripe group of 48 drives. The stripe group of 48 drives significantly outperformed the traditional method.

A comparison of Iometer results revealed a 44% improvement in the number of disk I/Os per second when switching from traditional volumes to a 48-drive wide stripe group. That is an amazing performance difference, much more than the five to 15% difference that we found between iSCSI and FC.

Some iSCSI storage systems may not have well-tuned performance optimized iSCSI target drivers. This is the fault of the storage vendor and they need to go back to their R&D group and do a better job. Additionally, ESG Lab has found that using a TCP/IP offload engine (TOE) on the iSCSI target port within the storage system can have a measurable positive impact on performance. Some iSCSI storage systems do not have integrated TOE support.

The architecture of the storage system, the speed and number of processors, the amount of memory and the intelligence of its caching algorithms, the speed the disk drives and number of drives in a stripe group, the number of host ports and the backend interconnect all play a major role in performance. I recommend that you evaluate the storage system based on all of the above criteria. It is the storage system itself that will make a bigger difference. The speed of iSCSI is not the issue.

iSCSI: Why a VMkernel and a Service Console?

Adam — Wed, 28 Oct 2009 02:46:19 +0000

Does Vmware have Microsoft Vista properties? You know, we’ll just chuck it in at the end and pray it works.

Frantic calls to and from on Saturday night lead myself and a friend to the question, “What the hell is up with iSCSI?”

It seems that the initiator works only with a service console. In fact, the observed results showed that iSCSI was being run with the Service Console’s IP address DESPITE have a vmkernel port configured in the same networks as the iSCSI target.

Well, this is not entirely true. Vmware needs a Service Console to do the initial CHAP authentication and establishment and only then does it switch over to the configured vmkernel interface.

Why even bother with an iSCSI “interface” (vmkernel port0 if you’re going to initiate with the Service Console? In fact, more to the point, why does it split the authentication and actual iSCSI traffic across them.

Based on various forum posts, it sounds like iSCSI was a last minute throw in by VMware, and just not done completely right.

Hence, Vmware Vista.

REF: my mobile phone log with the lingfish and http://communities.vmware.com/thread/209361

Vmware ESX3.5 Templates

Adam — Mon, 28 Sep 2009 02:47:52 +0000

Templates and their usage are one of the big time savers in ESX. The ability to setup a “perfect” vritual machine and then deploy multiple copies of it allows us to minimise the risk of deployment errors. However, there are a few limits around what can be achieved.

Specifically, in the environment I am in at the moment, there is a requirement for the operating system to be deployed to one LUN and the any application partitons to be deployed to another LUN.

Templates only allow me to deploy the machine’s virtual hard disk to one LUN and not split across two. Hence the problem.

There is no quick fix that I could find; so it was off the scripting land I went.

First things first. The customer required the deployment of twenty seven virtual machines all based off one. The process is prety simple; open my Virtual Infrastructure Client, point it at a Virtual Center (ESX servers by themselves don’t have template functionality), and convert the baseline virtual machine to a template.

I could have chosen to deploy twenty seven virtual machines manually, but, well, I’m lazy. So, the lazy man’s solution? Scripting!

The process was as follows:

1) Remove the 40GB hard drive (cloning and templating doesn’t allow hard drives to be placed in multiple datastores eg. osLUN and appLUN)
2) Create a template of the virtual machine.
3) Install powershell and the Vmware VI Toolkit giving us the powershell for Vmware functionality.
4) Save the following to a .ps1 (Powershell script file):

$array = “Machine1″,”Machine2″,”Machine3″ (and so forth)
$TARGET = “FQDN_OF_TARGET_ESX_SERVER”

Connect-viserver FQDN_OF_VIRTUAL_CENTER

foreach ($vm in $array)
{
$vm=New-VM -Name $vm -Template BASELINE_TEMPLATE -Host $TARGET -Datastore osLUN
}

The above script will connect to the VI Center (it will prompt for username and password) and then create a new machine called “Machine1″ etc. for each machine in the list. It will place the virtual machine on the esx server in $TARGET and build it from the template called BASELINE_TEMPLATE.

Now comes the hard part. There are 27 42GB drives that need to be created for the virtual machines. There is no powershell that will allow this to happen. Therefore, a script is required.

In appLUN is a vmdk called BASELINE_TEMPLATE_VMDK and this is the virtual hard drive (pre partitioned and formatted) that will be used for the virtual machines.

for DESTINATION in machine1 machine2 machine3 machine4 ETC.
do
# Copy
echo “Copying $SOURCE to $DESTINATION”
mkdir $DESTINATION
cp -a $SOURCE/* $DESTINATION/
# Rename vmdk files
cd $DESTINATION
for file in $(ls -1 *.vmdk)
do
echo “Renaming $SOURCE vmdks to $DESTINATION”
DEST=$(echo $file | sed s/$SOURCE/$DESTINATION/g)
mv $file $DEST
done
# Edit .vmdk
for file in $(ls -1 *.vmdk | grep -v flat)
do
echo “Replacing references to $SOURCE in .vmdk file $file”
sed -i s/$SOURCE/$DESTINATION/g $file
done
cd ..
done
echo “Finished”

This will copy the BASELINE_TEMPLATE_VMDK virtual hard drive one time for each virtual machine and make sure it is useable for that machine. It will also go through the .vmdk file and change the reference from BASELINE_TEMPLATE_VMDK to the destination virtual machine name.

Once it is run, we have to manually add the newly created vmdk to the corresponding virtual machine.

Then it is a simple matter of logging into each virtual machine and changing their host names and IP addresses.

There is still a fair bit of manual intervention involved, but at least the boring, tedious bits (deploying the templates and copying the hard drives) has been automated.

If you can offer any other scripts that can complete the picture, let me know!

Minor Deviations

Adam — Thu, 27 Aug 2009 06:46:06 +0000

Okay, more than once I have deviated madly off topic in this blog, but this one is special, I promise.

Everyone loves Calvin and Hobbes, don’t they? If you’re not familiar with it:

Calvin and Hobbes was a syndicated comic strip written and illustrated by Bill Watterson, following the humorous antics of Calvin, an imaginative six-year old boy, and Hobbes, his energetic and sardonic stuffed tiger. The pair are named after John Calvin, a 16th-century French Reformation theologian, and Thomas Hobbes, a 17th-century English political philosopher. The strip was syndicated daily from November 18, 1985 to December 31, 1995. At its height, Calvin and Hobbes was featured in over 2,400 newspapers worldwide. As of the publication of The Complete Calvin and Hobbes, more than 30 million copies of the 17 Calvin and Hobbes books had been sold.

Set in the contemporary Midwestern United States in an unspecified suburban community, the broad themes of the strip deal with Calvin’s flights of fantasy and his friendship with Hobbes, his misadventures, his unique views on a diverse range of political and cultural issues and his relationships with the people in his life, especially his parents. The dual nature of Hobbes is also a recurring motif: Calvin sees Hobbes as a live anthropomorphic tiger, while other characters see him as a stuffed animal. Though the series does not mention specific political figures or current events, it does mention broad issues like environmentalism, public education, and the flaws of opinion polls.

That’s a LOT of comic books!

I have taken it upon myself to get all Calvin And Hobbes strips without duplicates. After only a few minutes research, here is the list I have discovered:

•The Essential Calvin and Hobbes
•The Authoritative Calvin and Hobbes
•The Indispensable Calvin and Hobbes
•Attack of the Deranged Mutant Killer Monster Snow Goons
•The Days are Just Packed
•Homicidal Psycho Jungle Cat
•There’s Treasure Everywhere
•It’s A Magical World

The other option, of course it to buy the Collected Works!

vSphere multi-pathing failover

Adam — Tue, 25 Aug 2009 05:45:18 +0000

Whilst researching some storage options for a client, I stumbled across this pretty useful information

1) How often ESX checks for path failure?
==> As soon as an I/O request to a path fails, ESX will initiate a path failover. If there is no I/O outstanding to a path. ESX will probe each physical path every 5 minutes, by default, to proactively detect path failure.

2) How long will ESX wait before trying a different storage path?
==> A different storage path is tried immediately.

3) What happens between the failure detection and the connection recovery?
==> I/O requests will be queued.

4) Under which circumstances will a host initiated target reset or LUN reset occur?
==> Resets are not typically initiated by ESX. Two exceptions to this rule are:
- if a path failure occurs while there is a SCSI-2 reservation outstanding on the failed path
- if the VM or userworld that initiated the I/O request sends a request to abort the outstanding I/O request.

5) Will ESX ever force a LUN trespass in the array?
==> Only in the case of an A/P array when there is no working path on the same SP as the failing path. For example, when using an EMC Clariion (which uses the “trespass” command).

Technically point two needs some work.

ESX SCSI layer has a latency between when the failure occurs and when it will try a different path. This is usually determined by when the device driver returns a failure ie. I/O errors. Supported fibre channel devices will do this inside of thirty seconds. The ESX SCSI layer then has thirty seconds to use another working path. On an active/passive array the activate/trespass/etc command can take a few seconds to complete.

Within a virtual machine, the registry ([HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Disk]
“TimeoutValue”=dword:000000be) can be configured to wait before issuing an abort for an outstanding I/O. If this key is enabled

ESX states that within this 60 seconds it will: - detect that a path has failed - select a new path - activate the new path - re-issue the cmd from the guest to the new path - have the newly issued cmd complete successfully and return to the guest.

Vmware File Manager

Adam — Sun, 05 Jul 2009 05:25:11 +0000

There are a number of ways to upload and download files to and from the actual ESX server in my environment. The least favourite one I have would have to be browsing the datastore via VI Client!

This is the sort of problem that can happen, especially if you’re trying to download a file from a running Virtual Machine.

The error that appears is a:
Expected put message. Got: ERROR

The problem here is file locks. Old log files (no longer in use) will have no locks on them, so can be downloaded. Files like the vswp (Virtual Machine Swap) will be in constant use by the virtual machine and so cannot have the locks brokent to allow them to be downloaded.

The simple solution is to turn off a vritual machine before downloading the files relating to it.

Distributed File Server (DFS) Reporting

Adam — Mon, 29 Jun 2009 03:19:05 +0000

Well, haven’t I been out of the loop for a while?! Thankfully, my unemployment lasted nowhere near as long as I would have expected in the harsh economic climate and I am adjusting brilliantly to corporate/enterprise life once again (like there would be any doubt!)

All bragging aside, I have been working on a Storage Craft disaster recovery (DR) solution combining Vmware and Microsoft Distributed File System. The basic structure is this:

Storagecraft takes snapshots/incremental backups of virtual machines in the existing infrastructure and places them into a folder on it’s local storage that is then replicated across the WAN to the DR site. The DR site, also a Storagecraft server and also having a standalone ESX 3.5 server, can restore images of these machines with a fifteen minute tolerance of when the machine failed. This is all reliant on the most recent incremental backups being transmitted across the wire successfully.

Of course, as a DR site, the client is extremely interested in as much reporting as possible and this includes DFS. There is a builtin reporting function in DFS that requires manual intervention to trigger. The client requires automated reporting, as you would expect.

After trawling the internet, I found a workable solution combining a script and task scheduling in Windows 2003 R2.

As an aside, there was this handy little reference that I stumbled across as well. Nothing too surprising, but definitely worth thinking about.

The solution for the reporting was a script ripped from:

http://blogs.technet.com/filecab/articles/437214.aspx

and it looks like this:
####### ## REM This script needs to run as a daily scheduled task by someone who has ## REM local administrator rights for all machines in reported RG. This script ## REM will generate a health report for all members of the replication group, ## REM with backlog based on a specified reference member. When reports are ## REM complete they will be saved to a specified share. When ALL reports are ## REM done, mail will be sent to the specified e-mail alias with links to the ## REM share. #######


@echo off
set CURRDATE=%TEMP%\CURRDATE.TMP

set CURRTIME=%TEMP%\CURRTIME.TMP
DATE /T > %CURRDATE%

TIME /T > %CURRTIME%
######

## REM This adds the date and time to the health report name and to the title

## REM of the e-mail.

######
set PARSEARG="eol=; tokens=1,2,3,4* delims=/, "

for /F %PARSEARG% %%i in (%CURRDATE%) Do SET DDMMYYYY=%%j-%%k-%%l
set PARSEARG="eol=; tokens=1,2,3* delims=:, "

for /F %PARSEARG% %%i in (%CURRTIME%) Do Set HHMM=%%i%%j%%k
#######

## REM Here you define the different resources by specifying the name of each

## REM replication group, server name and share name.  You can do this for as

## REM many replication groups as you want, but keep in mind that the script

## REM will wait until all reports complete before mail is sent out.

#######
set RG1_Report=\\Server_name\Share_Name\RG1_Name-%DDMMYYYY%-%HHMM%.html

set RG2_Report=\\Server_name\Share_Name\RG2_Name-%DDMMYYYY%-%HHMM%.html
@echo on
#######

## REM This generates the reports with Dfsradmin.exe (you can use help to see

## REM what else it can do.)  You need to set the name of the replication

## REM group, the name of the reference server for calculating backlog, the

## REM domain the RG is in, and a name for the report.  The time and date

## REM variables we defined earlier in the script

#######
dfsradmin health new /rgname:RG1 /refmemname:Domain_name\Server_Name /domain:domain_name.com /ReportName:%RG1Report%

dfsradmin health new /rgname:RG2 /refmemname:Domain_name\Server_Name /domain:domain_name.com /ReportName:%RG2Report%
echo RG1  %RG1Report% >  %TEMP%\healthMessageBody.txt

echo RG2  %RG2Report% >> %TEMP%\healthMessageBody.txt
#######

### REM You will need to obtain an email command line application for this

### REM next part.  You will use the program to send an email with links to

### REM the health reports to an alias of your choice.  You may want address

### REM this mail to a distribution group.

### REM

### REM The command below is an example of how to send the mail using a

### REM ficticious command line mail application.  You will need to reformat

### REM the command to be compatible with your command line mail application.

### REM

### REM You many need to give the sending alias the "Send As" right

#######

cl_mail.exe smtp server:my_smtphost to:some_Alias@domain.com from:your_alias@domain.com subject:"DFS Replication Health Reports %DDMMYYYY%" message:@%TEMP%\healthMessageBody.txt auth:sspi

The only potential problems are:

The requirement for a command line email utility
Only notification of the report’s location, not the report itself
Size over time. Each report generated takes about 60KB of disk space.
The format of the report ie. XML/HTML

Still, it’s better than nothing!!

The HA “split brain” problem

Adam — Wed, 27 May 2009 03:28:41 +0000

This ISN’T something that occurs frequently, but it is something that I would want to test in HA (High Availabilty) before I went to production!

Sometimes when you set the DAS Isolation addresses, it fails to work, ie. when you take vswif0 offline, the virtual machines get shutdown anyway. Given this is emulating an “isolation respone”, where the service console ONLY fails to respond to the network, the virtual machines should only shutdown if the other cards in the server fails to respond.

Ideally, I want my DAS Isolation addresses in a totally seperate subnet from my service console. When I set up another Service Console, it’s always a good idea to remove HA from the cluster, wait for everything to settle down and then re-enable it on the cluster (a couple of minutes work). This makes sure both service consoles are sending heartbeats.

We can view the heartbeats via tcpdump and monitoring port 8044 by using the following commands:

# tcpdump -i vswifX port 8044 # tcpdump –i vswifY port 8044

As a final check, have a look at /var/log/vmware/aam/aam_config_util.def for the isolation addresses. It should look something like:

Start Object esx5 nodefd esx5 { nodeAddrs = { { sourceType = isolation source.addr = destination.addr = 172.16.0.36 } { sourceType = isolation source.addr = destination.addr = 10.0.0.52 } { sourceType = domain source.addr = 172.16.0.105 destination.addr = 224.0.6.127 } { sourceType = domain source.addr = 10.0.0.5 destination.addr = 224.0.6.127 } } }

End Object esx5

Also, if das.usedefaultisolationaddress is not set to false the service console’s default gateway appears as an isolation address in the /var/log/vmware/aam/aam_config_util.def file in addition to any ip address configured with das.isolationaddress and das.isolationaddress2.

Note: There is an email link embedded within this post, please visit this post to email it.

New Vmware Convertor 4.0.1

Adam — Mon, 25 May 2009 04:15:13 +0000

Vmware Convertor allows us to P2V (Physical to Virtual) migrations. That is, taking (potentially) a running machine on my network and importing it directly into my virtual infrastructure.

So, whilst vSphere is the new kid on the block, Vmware have also released an updated version to this genuinely brilliant utility!

What’s New
The VMware vCenter Converter Standalone 4.0.1 release is an update to Converter Standalone 4.0, and includes the following new features:

Support for vSphere 4.0 as source and destination targets:

Support for configuring target disks as thin provisioned disks

Support creation of IDE disks on vSphere 4.0

Support for backup products to restore vSphere 4.0 virtual machines backed up using VCB

Support for creation of virtual hardware version 7.0 virtual machines on vSphere 4.0. targets as well as migration of hardware version 7.0 virtual machines from Workstation and Server platforms to vSphere 4.0

Support for importing OVF 1.0 single virtual machine images.

Support for customization of Windows Server 2008 guests.

Note: There is an email link embedded within this post, please visit this post to email it.