For quite a while there has been confusion over how VMware’s Transparent Page Sharing (TPS) feature works with vSphere 4 running on Nehalem (or other modern) processors. Many people were noticing that it appeared that TPS was not actually working anymore and looked for ways to fix the problem.

In my recent post on the effects of ASLR in vSphere the comments turned into a discussion about TPS on modern processors. And there are countless posts about this issue on the VMTN forums where folks are looking for a fix. In reality nothing is broken and there is no need to fix the issue.

VMware has published a KB article that gives more information on TPS with Nehalem processors and why it appears TPS isn’t working (this affects modern AMD processors also). The short version is that TPS uses small pages (4K), and Nehalem processors utilize large pages (2MB). The ESX/ESXi host keeps track of what pages could be shared, and once memory is over-comitted it breaks the large pages into small pages and begins sharing memory.

Many people think this is a bug in ESX that needs to be fixed. This likely started because when vSphere 4 was released there was a bug around memory usage on ESX hosts with Nehalem processors. In reality the bug was that vCenter was triggering high memory usage alarms for virtual machines running in this configuration. Nothing was actually wrong but because the host was using all of the assigned memory for the VM, vCenter was incorrectly triggering the alarm. That behavior has since been fixed with a patch and is no longer an issue.

So what does this actually look like? When a VM is powered up on an ESX host with Nehalem processors, the amount of host memory in use will not drop down as the VM uses less memory or becomes idle. Those of us that have been using ESX for a long time likely found this scenario disturbing.

From vSphere Client (red highlighted section shows guest taking all of the 2GB assigned memory, yet memory usage in the guest is very low):

From esxtop (red highlighted section shows almost no memory being shared with page sharing):

The above screenshots show a host that is under-committed on memory and so no page sharing is occurring.  If the host gets over-commited page sharing kicks in automatically by breaking up large pages into small pages.  You can force the use of small pages on all guests all the time by changing the value of the advanced option Mem.AllocGuestLargePage to 0.  I don’t really see any reason to do this – remember that TPS isn’t broken and what you see in the above screenshots is normal and expected.

Once host memory is over-committed (or if you use the advanced option), memory sharing kicks in and things look like they normally do when page sharing is taking place.

From the vSphere Client (red highlighted section shows guest taking very little of the assigned 2GB memory as page sharing has kicked in):

From esxtop (red highlighted sections show a large amount of shared memory and the host is over-commited on memory by 48%):

A quick note on the esxtop screenshot above – it was taken from a VDI environment where all workloads are identical so that explains the high amount of shared pages.  It was also overcommitted more than normal as it was taken during host maintenance.

I hope this clears up some of the confusion around TPS on modern Intel/AMD processors.  In short, don’t get hung up on the fact that TPS isn’t kicking in like it did with older processors.  Nothing is broken, TPS is working as expected, and it will kick in when you actually need it.

In Q4 last year, Citrix made its NetScaler physical appliances available as a virtual appliance.  Labeled as the “VPX”, the full featured virtual iteration of the appliance dropped its price point and made it more accessible to SMB customers.  Citrix has now made the Access Gateway (CAG) and Branch Repeater physical appliances also available as VPXs.  At this point, Citrix has made three of their ‘core’ Networking products available as VPX appliances, which are recapped below.

Citrix NetScaler VPX

• Released last year in Q4.  Initially, this virtual appliance was only released for XenServer, but now has full support for ESX.  Expected to support Hyper-V in the late summer.
• Licensed by throughput and available in 10 Mbit, 200 Mbit and 1000 Mbit flavors.
• Free “Express” edition with 1 Mbit throughput limitation available for lab and PoC environments.
• Standard NetScaler “Advanced”, “Enterprise” and “Platinum” flavors available.

Citrix Access Gateway VPX

• Released earlier this month.  Currently only supports XenServer. Support for ESX and Hyper-V expected in the next 6-12 months.
• Provides same feature as Model 2010 Access Gateway physical appliance.
• Free “Express” edition exists that provides access to 5 concurrent users that are valid for 12 months.

Citrix Branch Repeater VPX

• Released in Q1 of 2010.  Currently only supports XenServer. Support for ESX and Hyper-V expected in the next 6-12 months.
• VPX Appliance does not support the following features available in the physical appliance:
  • Group Mode
  • Ethernet bypass card
• Still requires Citrix Repeater appliance, which is not available in VPX format.

This week at Citrix Summit/Synergy, Citrix finally revealed details behind their much anticipated client (bare metal) hypervisor.  To recap, for the folks who are not following, this will finally bring “offline VDI” to XenDesktop.  It will also match (and potentially beat) VMware’s current offline VM checkin/check out functionality currently available in View.

XenClient 1.0 will be released later this week for download on MyCitrix and is being demoed and talked about at the conference.  After playing with it at one of the demo stations and talking with some Citrix Engineers, here are some details:

• Unlike VMware’s View, XenClient is a Type 1 hypervisor.  This means it lives above the client side OS (Windows).  Once installed, the user has the option to boot into whatever VMs are available on the laptop.
• At the moment, XenClient will only support a small subset of hardware types.  This includes Dell’s Latitude E series, Dell Optiplex 780, and  HP EliteBook laptops.  Full HCL to be published later in the week.
• Citrix Synchronizer is the server appliance that chats with the XenClient (over HTTP/SSL) to continuously sync and update the local running VM back to the Data Center.
• Synchronizer will be available as a virtual appliance running on XenServer.  According to Citrix, there is no planned version for ESX.  – I am sure this will change though.
• XenClient can be installed as a standalone or in conjunction with “Synchronizer”.
• XenClient supports paravirtualization to allow VMs direct access to hardware (using it’s native driver).  For example, a VM under XenClient can tap directly into a GPU for accelerated video playback and graphic intensive applications.  – This demoed very well with the engineer playing back an HD video file without skipping within a VM.
• At the moment, only a small subset of USB devices are supported through XenClient.

I am at Synergy all week, so I expect to learn more details about XenClient over the course of the week.  If you guys have questions, feel free to post them in the comments and I’ll try to get those questions answered while at the conference.

When VMware released vSphere 4 last year, one of the changes they made was a completely re-written software iSCSI initiator. This was done to optimize performance which is great considering how popular iSCSI SANs have become. They also gave the ability to use Round Robin MPIO (mutlipathing) in the software initiator in addition to Fixed Path and MRU which were previously available.

I’m working on a vSphere implementation using Dell EqualLogic SANs and wanted to configure Round Robin on all of my datastores. Dell has a great whitepaper on how to set this up, but unfortunately the document fails to mention one key thing: this doesn’t change the default path selection plugin (PSP) from Fixed to Round Robin.   That means that you’ll have to set the multipathing policy to Round Robin on all of your existing datastores and will have to remember to do that on all future datastores. When you’ve got multiple ESX hosts with lots of  datastores this can quickly become a pain.

Luckily there is a way to force the default multipathing policy to Round Robin. The following commands can be used to change the default PSP to Round Robin as well as configure round robin specifically for the EqualLogic provider.  These commands can be entered at the Service Console or via the vSphere CLI 4.0:

esxcli nmp satp setdefaultpsp –satp VMW_SATP_DEFAULT_AA –psp VMW_PSP_RR
esxcli nmp satp setdefaultpsp –satp VMW_SATP_EQL –psp VMW_PSP_RR
esxcli corestorage claimrule load
esxcli corestorage claimrule run

Note that “satp” and “psp” are preceded by two dashes and not a single dash as it appears in this blog post.

Once you enter those commands (no rebooting required) any volume you add, either new or existing, will use Round Robin MPIO by default.

I’ve seen a lot of talk lately about VMware’s Transparent Page Sharing (TPS) and how it is affected by ASLR in Windows 2008/Windows 7. I wanted to see if there was any real measurable reduction in shared memory when using ASLR vs. when it was disabled. First, let’s talk about what TPS and ASLR actually are and what the acronyms mean.

Transparent Page Sharing is a technology built into ESX/ESXi that looks for identical guest memory pages and writes them to memory just once. Guests can then share those identical pages rather than each writing the same page to memory. TPS is a great feature that allows for memory overcommittment, especially on hosts that run many of the same type of workload.

Address Space Layout Randomization (ASLR) is a security feature that randomizes the position of data in memory, making it more difficult for attackers to predict where data can be found while in memory. This feature has been enabled in Windows since Windows Vista, and other operating system such as Linux and MacOS implement this in some form as well.

Since ASLR randomizes information in memory it makes sense that it would be more difficult for TPS to find identical memory pages and thus memory sharing would be reduced. But just how much of a difference does it make? I decided to try and find out. Here are the specs from my test environment:

Server: HP DL385 G1 (AMD Opteron 275)
ESX: 4.0.0 build 244038
Guest OS: Windows Server 2008 R2
Guest RAM: 2.5GB

All guests were cloned from the same template and have the same software installed. On guests TESTSRV1 and TESTSRV3, I left the default settings. On TESTSRV2 and TESTSRV4, I disabled ASLR using the following regkey:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management]
“MoveImages”=dword:00000000

In all of my testing, including leaving VMs idle and also running memory tests, I found no measurable difference in the amount of memory shared with TPS. I also looked at esxtop to see how much memory was actually being shared and I saw virtually no difference whatsoever between VMs that had ASLR enabled and those that had it disabled.

Host Memory Usage:

esxtop statistics:

The SHRD and SHRDSVD columns represent how much memory is being shared with TPS and the total memory savings. Clearly there is quite a bit of memory sharing going with or without ASLR enabled.

Why would this be the case since it makes sense that TPS would be hurt by ASLR? ASLR requires applications to “opt-in” to have their memory randomized, and I suspect that much of Windows 2008 R2 is not opted in. Perhaps applications will come out in the future that are written to take advantage of ASLR, but at the moment that doesn’t appear to be the case.

Of course this is by no means a definitive test as it wasn’t run with production systems and real users running real applications. That said, I think it shows that ASLR does not dramatically reduce the amount of memory shared with TPS. I did also look at production systems left at the default settings (ASLR enabled) and saw similar memory sharing gains. I’m curious if others have seen similar results in their environments, so drop me a line if you’ve done any similar testing.

More info:

What is ASLR (Wikipedia)
Interpreting esxtop statistics

For years organizations have relied on tape drives and changers for backup and recovery of their critical data. Despite many predictions to the contrary, tape is still alive as we begin 2010.

When virtualization became popular it presented a challenge to those looking to continue to use their tape drives in fully virtualized environments. If you were using VMware you could use SCSI pass-through to present a tape drive or changer directly to a virtual machine but that prevented you from using any advanced features like VMotion. It also tied your tape drive and VM to a single host containing a SCSI card, making things complicated if that host were to experience a hardware failure.

While this is still possible in vSphere 4 (and the previous version), this configuration is not ideal.  Instead, consider converting that SCSI tape changer into an iSCSI target that can be used on any virtual machine attached to any host by using an iSCSI-to-SCSI bridge.  These bridges let you attach your tape changer directly to the device and then present the tape changer to virtual machines as an iSCSI target.  There are several different vendors providing this technology, including Atto Technology, Paralan, and others.

Once the tape drive is attached to the iSCSI bridge and configured as a target, you simply use the Microsoft iSCSI initiator inside a virtual machine to connect to the device.  The tape device will appear to the virtual machine as if it were any other iSCSI target (like a SAN LUN).

After connecting to the target in the iSCSI Initator, the tape device will become visible in Device Manager on the VM and within tape backup software.

After the tape device has been successfully discovered, the virtual machine can then be managed with features like VMotion, HA, and DRS because the VM won’t be tied to an individual host.  This configuration also opens up other design possibilities, such as multiple backup servers running different backup products.  Using the iSCSI bridge provides a lot more flexibility than directly attaching the tape device to an ESX host.

Sounds great, right?  As always, there are things to consider before moving forward with this type of solution:

1) Does your backup software vendor and tape changer vendor support this setup?

2) Will this setup meet the performance requirements of your environment?  In practice I’ve seen these devices push 2 GB/min or more, similar to the performance of direct attached tape devices.

Using an iSCSI-to-SCSI bridge opens up a lot of possibilities for keeping a tape device in your fully virtualized environment.  It also simplifies your setup and allows you to take advantage of enterprise features of your virtualization product.  Finally, for around $1,200 for the iSCSI bridge this is also an affordable solution to a common problem.

High CPU utilization when screensaver kicks in

That ended up being around 900MHz of a modern Intel X5460 CPU.  If this was lunch time and 15 people left their VDI session idle, this easily could have caused high CPU utilization across the entire ESX host and hurt performance for everyone.

Most organizations require that a screensaver kick in after a set period of idle time to protect access to the desktop.  A much better alternative is to use the Blank screensaver available in Windows XP/Vista/7 to protect the screen and require a password to unlock.  You get the protection without the unnecessary drain on the CPU.

My last post Citrix Provisioning Services Part 1 – What Is It? served an introduction to what exactly Citrix Provisioning Services is capable of. Below I hope to open people’s eyes to using PVS for something other than VDI, as it is often thought of as a part of the XenDesktop suite. However PVS is actually independent of XD or VDI, and can be utilized in combination with XenApp to bring single-image benefits to the Terminal Services world.

Provisioning Services allows for server consistency, easier maintenance, dynamic servers, and aids in disaster recovery.

• Consistency – As a best practice every XenApp server delivering the same applications should be 100% identical to the rest of the farm. However, obtaining this is easier said than done.  By streaming the same image to every server, each server is inherently and 100% the same as the rest.
• Maintenance – Updating and patching large farms can be a very time consuming task, and anything done to one server must be repeated for the entire farm to maintain consistency. With PVS, patches and software installations are applied once to the master image and on next reboot, each XenApp server boots the new updated image. In addition to software patching and installation, Terminal Servers need to be completely refreshed periodically to keep them clean and performing optimally; they are used by dozens of different users, reducing performance and resulting in inconsistent servers. A typical server refresh requires the server to be re-imaged and the software redeployed, a time consuming process that can be prone to error, leaving a server in an unusable or inconsistent state. Operating system streaming with PVS results in a completely fresh and optimized server on every reboot.
• Dynamic – PVS allows for a dynamic XenApp farm instead of a static one. As load rises and additional servers are needed, they can be quickly brought online in seconds instead of hours. Conversely, as load drops, un-needed servers can be powered off or repurposed as needed. A server becomes a vessel for different workloads and can be a XenApp server one day and an IIS server another if need be. Since Provisioning Services is capable of streaming to both physical and virtual servers, administrators have the ability to utilize different types of resources all from the same master image(s).
• Disaster Recovery – Creating a disaster recovery plan for the XenApp environment often requires complex processes, scripts and configurations. Assuming a PVS server has been built in the DR site, and that the master image has been replicated as well, quickly bringing an entire farm of XenApp servers online becomes a simple task.

Creating a XenApp environment that is more dynamic and easier to maintain is a goal for many XenApp administrators. The addition of Provisioning Services to a XenApp implementation can go a long way to achieving those goals. By leveraging the single-image management capabilities of PVS, administrators can dramatically reduce the costs involved with deploying and maintaining their XenApp farms. While at the same time, guaranteeing consistency between and ensuring peak performance of each server in the farm. All while being capable of quickly adapting to changes in load and disaster scenarios.

One of the great features of desktop virtualization (VDI) being touted by the industry is the ability to manage and update all of your desktops from a single central master image.

Citrix’s solution to the single image process is accomplished by a product called Provisioning Services (PVS). This software is the result of their purchase of a company called Ardence back in 2007. Provisioning Services is an often misunderstood piece of software, and its great benefits and potential are not necessarily apparent to everyone.

PVS works by streaming a master (read-only) image from the server to a target server or workstation. Any subsequent writes are then sent back to the PVS server and written to a cache file. The reads and writes are sent back and forth between the PVS server and target in a constant stream over the network. The easiest way to grasp this is to imagine that the cable connecting the hard disk inside of the server to the motherboard (and thus the CPU and RAM) is replaced by a network cable running back to the PVS server. The operating system sees the PVS disk as though it were a normal hard disk, and everything is done entirely transparent to the OS. The magic happens when the server is powered up; instead of booting from a local disk it is instead set to boot to the network card (PXE, BOOTP) which talks to a service on the PVS server, which streams the assigned operating system image to the target. The target device starts up immediately, as though it was booting from a local disk.

The beauty here is that this single read-only image can be simultaneously streamed to multiple diskless targets, both physical and virtual. This central image can now be maintained in one place. This makes tasks such as installing updates or new software quick and easy. After installing an update into the master image, all machines running that image will boot up into the updated image on next restart. To put that in perspective, think of the time and effort required to push out something such as a service pack to Windows or Microsoft Office to your entire firm. Now imagine simply installing that update once and having every machine in your environment receive that update on next reboot, without any additional effort.

Look for a follow-up post discussing the benefits that Provisioning Services can bring to a XenApp implementation.

With the release of VMware vSphere 4, VMware has released a very powerful management tool called Fault Tolerance (FT).  At a basic level, FT allows you to keep two virtual machines (a Primary VM and a Secondary VM) running in lockstep on two different physical ESX hosts.  If one of the ESX hosts were to experience a hardware failure, the VM protected with FT would remain running on the second host without any downtime.  This can greatly reduce downtime due to hardware failures and provide increased service levels for important applications.

FT is often compared to Microsoft Windows Failover Clusters, formerly Microsoft Cluster Server (MSCS), and in fact many have talked about how FT can replace Microsoft clustering altogether.  Rather than jump to conclusions like this, it is important to understand the use cases for both technologies.  In addition, there are several limitations to FT that need to be considered. Here are some important points to remember about FT:

1) FT only supports a single vCPU, limiting its usefulness for some applications (this will probably change in the future).

2) FT is meant to protect against host level failures only, such as physical server failures.

3) FT keeps protected virtual machines in complete lockstep, meaning whatever happens on the Primary VM also happens on the Secondary VM.  Why is this important to understand?  Guess what happens when the Primary VM bluescreens.

4) FT VMs share the same virtual disk file, meaning a storage level failure affects both.

Microsoft Failover Clustering, on the other hand, can help protect against application and operating system level failures in addition to physical server failures.  Clusters also make it easier to patch the underlying operating system with minimal downtime.  Finally, FT has a limited set of hardware that it is compatible with (see this link to check if your system is compatible) so that may limit its usefulness for some organizations using older hardware.

All of that said, FT is a great feature and definitely has a place in your virtual infrastructure.  Best of all it is available in vSphere Advanced and above, making it an affordable feature that many organizations may already own.  So what are good use cases for FT over Microsoft clusters?  Here are some thoughts:

• Provide hardware level protection to applications that don’t natively support any clustering functionality.  There are many examples here – web servers, application servers, etc.
• Provide hardware level protection to applications where clustering support may be available but is either expensive or requires a special license.  Document management servers, indexers, etc., may make good candidates here.
• Provide extra protection against hardware failures for critical applications during specific business periods where downtime simply cannot be tolerated, such as accounting servers during end of month processing.
• Your specific virtual infrastructure configuration precludes you from using Microsoft clusters due to Microsoft or VMware supportability.

VMware FT is a great new feature that can help provide extra uptime to virtual machines in your environment.  It is available in most versions of vSphere and should definitely be considered as part of a virtual infrastructure design.  Just make sure you understand the use cases for it and don’t rule out Microsoft clusters where they are appropriate.

See also:

FT Frequently Asked Questions