Quite a while back I saw that Eric Sloof had figured out how to add his Twitter feed directly into the VI Client.  I thought it was clever but didn’t really give it much more thought than that.

Today I decided to take that concept and extend it to systems that you might manage alongside your VI3/vSphere environment.  Storage management seemed like the obvious first choice.

I created an XML file called EqualLogic.xml in C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Plugins\SAN Management.  The contents of the file are as follows (you would replace the <url> section with the IP or DNS name of your SAN):

<scriptConfiguration version=”1.0.0″>
<key>EqualLogic</key>
<description>EqualLogic SAN Management</description>
<view parent=”Inventory.HostSystem”>
<title locale=”en”>EqualLogic</title>
<url>http://10.1.97.30</url>
</view>
</scriptConfiguration>

This gave me a nice way to manage my SAN from the same interface that I use to manage my vSphere environment.  It is simply opening a browser window within the vSphere Client and letting me manage the SAN.

The code above will make the EqualLogic tab visible only when clicking on an ESX/ESXi host.  If you wanted to extend that to other objects, you can simply adjust the “<view parent=” section.  For example, to also make this available at the cluster level you would include the following:

<view parent=”Inventory.Cluster”>
<title locale=”en”>EqualLogic</title>
<url>http://10.1.97.30/</url>
</view>

Similarly you could add Inventory.Global, Inventory.VirtualMachine, Inventory.Datacenter, etc.

This is a really simple way to make it easy to manage any web interface (not just EqualLogic) from within the vSphere/VI Client.  It’s not a new trick and has been out there for a while but I had never used it for this until today.

For quite a while there has been confusion over how VMware’s Transparent Page Sharing (TPS) feature works with vSphere 4 running on Nehalem (or other modern) processors. Many people were noticing that it appeared that TPS was not actually working anymore and looked for ways to fix the problem.

In my recent post on the effects of ASLR in vSphere the comments turned into a discussion about TPS on modern processors. And there are countless posts about this issue on the VMTN forums where folks are looking for a fix. In reality nothing is broken and there is no need to fix the issue.

VMware has published a KB article that gives more information on TPS with Nehalem processors and why it appears TPS isn’t working (this affects modern AMD processors also). The short version is that TPS uses small pages (4K), and Nehalem processors utilize large pages (2MB). The ESX/ESXi host keeps track of what pages could be shared, and once memory is over-comitted it breaks the large pages into small pages and begins sharing memory.

Many people think this is a bug in ESX that needs to be fixed. This likely started because when vSphere 4 was released there was a bug around memory usage on ESX hosts with Nehalem processors. In reality the bug was that vCenter was triggering high memory usage alarms for virtual machines running in this configuration. Nothing was actually wrong but because the host was using all of the assigned memory for the VM, vCenter was incorrectly triggering the alarm. That behavior has since been fixed with a patch and is no longer an issue.

So what does this actually look like? When a VM is powered up on an ESX host with Nehalem processors, the amount of host memory in use will not drop down as the VM uses less memory or becomes idle. Those of us that have been using ESX for a long time likely found this scenario disturbing.

From vSphere Client (red highlighted section shows guest taking all of the 2GB assigned memory, yet memory usage in the guest is very low):

From esxtop (red highlighted section shows almost no memory being shared with page sharing):

The above screenshots show a host that is under-committed on memory and so no page sharing is occurring.  If the host gets over-commited page sharing kicks in automatically by breaking up large pages into small pages.  You can force the use of small pages on all guests all the time by changing the value of the advanced option Mem.AllocGuestLargePage to 0.  I don’t really see any reason to do this – remember that TPS isn’t broken and what you see in the above screenshots is normal and expected.

Once host memory is over-committed (or if you use the advanced option), memory sharing kicks in and things look like they normally do when page sharing is taking place.

From the vSphere Client (red highlighted section shows guest taking very little of the assigned 2GB memory as page sharing has kicked in):

From esxtop (red highlighted sections show a large amount of shared memory and the host is over-commited on memory by 48%):

A quick note on the esxtop screenshot above – it was taken from a VDI environment where all workloads are identical so that explains the high amount of shared pages.  It was also overcommitted more than normal as it was taken during host maintenance.

I hope this clears up some of the confusion around TPS on modern Intel/AMD processors.  In short, don’t get hung up on the fact that TPS isn’t kicking in like it did with older processors.  Nothing is broken, TPS is working as expected, and it will kick in when you actually need it.

When VMware released vSphere 4 last year, one of the changes they made was a completely re-written software iSCSI initiator. This was done to optimize performance which is great considering how popular iSCSI SANs have become. They also gave the ability to use Round Robin MPIO (mutlipathing) in the software initiator in addition to Fixed Path and MRU which were previously available.

I’m working on a vSphere implementation using Dell EqualLogic SANs and wanted to configure Round Robin on all of my datastores. Dell has a great whitepaper on how to set this up, but unfortunately the document fails to mention one key thing: this doesn’t change the default path selection plugin (PSP) from Fixed to Round Robin.   That means that you’ll have to set the multipathing policy to Round Robin on all of your existing datastores and will have to remember to do that on all future datastores. When you’ve got multiple ESX hosts with lots of  datastores this can quickly become a pain.

Luckily there is a way to force the default multipathing policy to Round Robin. The following commands can be used to change the default PSP to Round Robin as well as configure round robin specifically for the EqualLogic provider.  These commands can be entered at the Service Console or via the vSphere CLI 4.0:

esxcli nmp satp setdefaultpsp –satp VMW_SATP_DEFAULT_AA –psp VMW_PSP_RR
esxcli nmp satp setdefaultpsp –satp VMW_SATP_EQL –psp VMW_PSP_RR
esxcli corestorage claimrule load
esxcli corestorage claimrule run

Note that “satp” and “psp” are preceded by two dashes and not a single dash as it appears in this blog post.

Once you enter those commands (no rebooting required) any volume you add, either new or existing, will use Round Robin MPIO by default.

I’ve seen a lot of talk lately about VMware’s Transparent Page Sharing (TPS) and how it is affected by ASLR in Windows 2008/Windows 7. I wanted to see if there was any real measurable reduction in shared memory when using ASLR vs. when it was disabled. First, let’s talk about what TPS and ASLR actually are and what the acronyms mean.

Transparent Page Sharing is a technology built into ESX/ESXi that looks for identical guest memory pages and writes them to memory just once. Guests can then share those identical pages rather than each writing the same page to memory. TPS is a great feature that allows for memory overcommittment, especially on hosts that run many of the same type of workload.

Address Space Layout Randomization (ASLR) is a security feature that randomizes the position of data in memory, making it more difficult for attackers to predict where data can be found while in memory. This feature has been enabled in Windows since Windows Vista, and other operating system such as Linux and MacOS implement this in some form as well.

Since ASLR randomizes information in memory it makes sense that it would be more difficult for TPS to find identical memory pages and thus memory sharing would be reduced. But just how much of a difference does it make? I decided to try and find out. Here are the specs from my test environment:

Server: HP DL385 G1 (AMD Opteron 275)
ESX: 4.0.0 build 244038
Guest OS: Windows Server 2008 R2
Guest RAM: 2.5GB

All guests were cloned from the same template and have the same software installed. On guests TESTSRV1 and TESTSRV3, I left the default settings. On TESTSRV2 and TESTSRV4, I disabled ASLR using the following regkey:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management]
“MoveImages”=dword:00000000

In all of my testing, including leaving VMs idle and also running memory tests, I found no measurable difference in the amount of memory shared with TPS. I also looked at esxtop to see how much memory was actually being shared and I saw virtually no difference whatsoever between VMs that had ASLR enabled and those that had it disabled.

Host Memory Usage:

esxtop statistics:

The SHRD and SHRDSVD columns represent how much memory is being shared with TPS and the total memory savings. Clearly there is quite a bit of memory sharing going with or without ASLR enabled.

Why would this be the case since it makes sense that TPS would be hurt by ASLR? ASLR requires applications to “opt-in” to have their memory randomized, and I suspect that much of Windows 2008 R2 is not opted in. Perhaps applications will come out in the future that are written to take advantage of ASLR, but at the moment that doesn’t appear to be the case.

Of course this is by no means a definitive test as it wasn’t run with production systems and real users running real applications. That said, I think it shows that ASLR does not dramatically reduce the amount of memory shared with TPS. I did also look at production systems left at the default settings (ASLR enabled) and saw similar memory sharing gains. I’m curious if others have seen similar results in their environments, so drop me a line if you’ve done any similar testing.

More info:

What is ASLR (Wikipedia)
Interpreting esxtop statistics

With the release of VMware vSphere 4, VMware has released a very powerful management tool called Fault Tolerance (FT).  At a basic level, FT allows you to keep two virtual machines (a Primary VM and a Secondary VM) running in lockstep on two different physical ESX hosts.  If one of the ESX hosts were to experience a hardware failure, the VM protected with FT would remain running on the second host without any downtime.  This can greatly reduce downtime due to hardware failures and provide increased service levels for important applications.

FT is often compared to Microsoft Windows Failover Clusters, formerly Microsoft Cluster Server (MSCS), and in fact many have talked about how FT can replace Microsoft clustering altogether.  Rather than jump to conclusions like this, it is important to understand the use cases for both technologies.  In addition, there are several limitations to FT that need to be considered. Here are some important points to remember about FT:

1) FT only supports a single vCPU, limiting its usefulness for some applications (this will probably change in the future).

2) FT is meant to protect against host level failures only, such as physical server failures.

3) FT keeps protected virtual machines in complete lockstep, meaning whatever happens on the Primary VM also happens on the Secondary VM.  Why is this important to understand?  Guess what happens when the Primary VM bluescreens.

4) FT VMs share the same virtual disk file, meaning a storage level failure affects both.

Microsoft Failover Clustering, on the other hand, can help protect against application and operating system level failures in addition to physical server failures.  Clusters also make it easier to patch the underlying operating system with minimal downtime.  Finally, FT has a limited set of hardware that it is compatible with (see this link to check if your system is compatible) so that may limit its usefulness for some organizations using older hardware.

All of that said, FT is a great feature and definitely has a place in your virtual infrastructure.  Best of all it is available in vSphere Advanced and above, making it an affordable feature that many organizations may already own.  So what are good use cases for FT over Microsoft clusters?  Here are some thoughts:

• Provide hardware level protection to applications that don’t natively support any clustering functionality.  There are many examples here – web servers, application servers, etc.
• Provide hardware level protection to applications where clustering support may be available but is either expensive or requires a special license.  Document management servers, indexers, etc., may make good candidates here.
• Provide extra protection against hardware failures for critical applications during specific business periods where downtime simply cannot be tolerated, such as accounting servers during end of month processing.
• Your specific virtual infrastructure configuration precludes you from using Microsoft clusters due to Microsoft or VMware supportability.

VMware FT is a great new feature that can help provide extra uptime to virtual machines in your environment.  It is available in most versions of vSphere and should definitely be considered as part of a virtual infrastructure design.  Just make sure you understand the use cases for it and don’t rule out Microsoft clusters where they are appropriate.

See also:

FT Frequently Asked Questions

In a previous post, I discussed how to expand virtual machine boot volumes with no downtime using Dell’s ExtPart utility.  Using this method is useful if you are using Windows Server 2000/2003, but is no longer required when using Windows 2008.  With Windows 2008 becoming the preferred server operating system going forward, using the method described below will become more and more common.

Microsoft has made it possible to expand boot volumes easily and on the fly without downtime without the need for any additional tools.  In this example, we’ll expand the C: drive of a Windows 2008 virtual machine from 25GB to 30GB.

1. Expand the size of the virtual disk.

2. Launch Disk Management by right clicking on My Computer, selecting Manage, and then selecting Storage\Disk Management.  If you do not see the unallocated space, right click on Disk Management and select “Rescan Disks.”

3. Right click on the currently allocated (blue) space, and select “Extend Volume…” to launch the Extend Volume Wizard.

4. Select Next at the Wizard launch screen, and then confirm the amount of space to add to the volume.

5. Select Finish to extend the volume to the new size.

Installing the VMware Tools package inside a VMware virtual machine improves overall performance and allows the use of advanced features and faster virtual hardware drivers.  The installation package also installs a tray icon that controls guest access to virtual hardware, time synchronization, etc.  Since most virtual machines are servers and end users don’t typically access the console of a server, worries about the security implications of leaving that tray application running have been fairly minimal.  However, as firms move towards solutions like virtualized XenApp servers or virtual desktops, this becomes more of a concern.

Removing administrator access to end users is unfortunately not enough.  For example, a user can open the VMware Tools tray icon and select the Devices tab, and from there can uncheck “NIC1″ and click Apply.  What happens?  You guessed it – the virtual NIC is disconnected and the user loses connection.  That’s bad in a virtual desktop environment since it will orphan the desktop and likely require a connection broker like XenDesktop to create another desktop but it is even worse on a XenApp server where the user potentially just disconnected dozens of other users as well.

This, and several other things found in the VMware Tools, can be dangerous to leave available to an end user even if they have no rights to the server itself.  To get around this, there are two approaches that make sense:

1) Remove access to the VMware Tools for end users.

2) Modify the VMX configuration file to prevent these actions.

I prefer the second method since it allows for more granular control over security, though if you’re interested in option one then you can read VMware’s KB article on the subject.  In order to prevent this at the VMX (virtual machine configuration file) level, simply add the following lines to the virtual machine(s) that you wish to protect (after powering it down):

isolation.device.connectable.disable = “true”
isolation.device.edit.disable = “true”

To see how to add one of these values to the VMX file via PowerShell and PowerCLI, it would look something like this:

$vm = Get-View (Get-VM NameofVM).ID
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmConfigSpec.extraconfig += New-Object VMware.Vim.optionvalue
$vmConfigSpec.extraconfig[0].Key=”isolation.device.connectable.disable”
$vmConfigSpec.extraconfig[0].Value=”true”
$vm.ReconfigVM($vmConfigSpec)

There are many other security parameters that can be set in the VMX file that are covered in VMware’s Security Hardening document (PDF).  The document covers this and many other common security best practices for virtual machines.  As always, test any change you make (especially the script above) before putting anything into production.

Microsoft has made a very interesting virtualization play this week by submitting source code for the Linux kernel. This will allow Linux guests to run on the maturing Windows 2008 hypervisor, Microsoft Hyper-V. This is an unexpected move from Microsoft to publicly release source code and an indicator of their appetite to compete with VMware’s dominate market share.

Microsoft stuns Linux world, submits source code for kernel

In an historic move, Microsoft Monday submitted driver source code for inclusion in the Linux kernel under a GPLv2 license. The code consists of four drivers that are part of a technology called Linux Device Driver for Virtualization.

Microsoft Linux Move Puts Pressure on VMware

Microsoft’s historic embrace of Linux technology could have wide-reaching impacts on the virtualization market and Microsoft’s rival VMware.

By allowing greater ability to run Linux on the Hyper-V virtualization platform, Microsoft is making a compelling case that it could be the virtualization vendor of choice for consolidation of Windows and Linux applications, says Gartner analyst George Weiss.

Microsoft has done a nice job of late regarding supporting their operating systems and applications when they are run in a virtual machine. First they created the Server Virtualization Validation Program to validate their software running on hypervisors from various vendors, including their own Hyper-V.  They’ve taken the SVVP one step further by adding a new tool called the Support Policy Wizard that makes it fast and simple to verify Microsoft support.  It can be found at the following link:

http://www.windowsservercatalog.com/svvp.aspx?svvppage=svvpwizard.htm

This tool makes it easy to select a specific configuration, including operating system, hypervisor, and Microsoft application, and then verify it against the SVVP to ensure Microsoft supportability.  It also lists any specific application features that are or are not supported.  Validating your configuration against Microsoft’s Support Policy Wizard is an important step, especially when virtualizing mission critical applications like Exchange or SQL.

Try the wizard and see the results for yourself.  I tried it myself and the results are below.  I selected Exchange Server 2007 SP1 running on vSphere 4 with Windows Server 2008 x64 and received the following supportability statement back:

Summary Support Statement*
This configuration is Supported.

* Customers with Premier-level support agreements should contact their account manager for more information
* Additional information is available in the “Support policy for Microsoft software running in non-Microsoft hardware virtualization software” which can be viewed here

Support Statement Details
Product: Exchange Server 2007 Service Pack 1 on VMware vSphere with Windows Server 2008 (x64) Guest OS

Search the Knowledge Base for information related to this configuration

For Exchange Service 2007 Pack 1 and later, see here for specific configuration information. Note: The Exchange Server guest virtual machine must be deployed on the Windows Server 2008 operating system.

Specific third-party virtualization information is available at here

To get support on the virtualization solution, the customer also needs to have a support agreement with the third party vendor.

Supported features: Anti-Virus, Back-up Software, Virtual Machine Management Software, Cluster Continuous Replication (CCR), Virtual Processors

Unsupported features: Unified Messaging, Dynamically Expanding Virtual Disks, Virtual disks that use differencing or delta mechanisms, Hyper-V Quick Migration combined with Exchange Clustering, Virtual Machine Snap Shots

Discovering that your ESX hosts have been unknowingly creating open snapshots can be an alarming, not to mention dangerous, event.

Third party storage and backup vendors frequently call the vCenter API to issue a snapshot creation of running virtual machines before they grab a copy of the virtual machine’s hard disk – VMDK files – which are located on the VMware VMFS datastore. This process is typically followed by an immediate deletion of the snapshot file which will merge the changes back into the initial VMDK file. Problems can occur when this process does not complete successfully leaving you with an “open” snapshot. This problem is compounded when it occurs multiple times against the same guest.

Each open snapshot can significantly degrade virtual machine performance and also contribute to poor storage utilization. Unless you monitor each of your virtual machines on a daily basis, you could quickly be left with an uncomfortable situation on your hands.

The script below can be easily customized for your environment and provide regular email reports for open snapshots.

#Initialize VI Toolkit

Add-PSSnapin VMware.VimAutomation.Core

#Configure VC credentials

$Username = ’service_account_name’

$password = ‘PASSWORD’

$server = ‘vcenter_host.domain.local’

#Setup email client

$SmtpClient = New-Object system.net.mail.smtpClient

$MailMessage = New-Object system.net.mail.mailmessage

$SmtpClient.host = “smtp_host.domain.local”

$MailMessage.from = “virtualcenter@domain.com”

$MailMessage.To.add(”admin@domain.com”)

$MailMessage.IsBodyHtml = 1

$MailMessage.Subject = “List of current VMware Snapshots”

#Get snapshots from all servers

Connect-VIServer $server -User $Username -Password $password

$Snaps = Get-VM | Get-Snapshot | Select VM,Name,Created

# Send email

$MailMessage.body = $Snaps | ConvertTo-Html

$SmtpClient.Send($MailMessage)

To execute the script, you will need to install Powershell and the VI Toolkit. Good luck!