I recommend reading Andre’s post to get more information on this topic.  The short version is this – Exchange/Outlook best practices do not necessarily work in a VDI environment.

When using Outlook in an Exchange environment, it is recommended to use Cached Exchange Mode.  In this mode a copy of the user’s mailbox is downloaded into an OST file and stored offline on the user’s desktop.  This mode offers better performance for the end user and reduces utilization on the Exchange environment as well.  In addition, using Cached Exchange Mode allows users to use Outlook Instant Search for fast searching of items in their mailbox.  Instant Search works by indexing the contents of the OST file so all searches occur locally and not on the Exchange server, further improving performance and reducing utilization on Exchange.

VDI environments that we see at our clients are typically configured as non-persistent or floating pools of desktops.  That is, each user connects to a pool of identical desktops and grabs whatever desktop is available.  When the user logs off, any changes written to the VDI desktop are discarded and the desktop returns to a pristine state.  There are mechanisms and tools in place to make sure user data is retained at logoff.

So if user data is retained at logoff, why can’t we use Cached Exchange Mode in non-persistent VDI environments?

• The OST file is equal in size to the user’s mailbox so storing a 15-30GB OST (not unusual at our clients) is not that practical from a performance or storage perspective.  If this data is being stored on the SAN, then you’re essentially doubling your Exchange storage (which may already be doubled or tripled if you’re using Exchange 2010 w/ DAGs).  In addition, the length of time it would take to download that file every time and the I/O impact that would cause makes it completely impractical.
• OST files are not supported when stored on network shares, so redirecting the OST to a home directory is out.
• Indexing of files on virtual desktops is typically disabled to reduce I/O demands.  This would prevent the use of Outlook Instant Search even if the OST was present.

For these and other reasons, Outlook is typically configured in Online mode when used with VDI.  This keeps all mailbox operations and searches on the Exchange server, placing the processing and I/O burden solely on the Exchange environment.  That sounds bad, but advances in Exchange technology specifically with Exchange 2010 have made this much less of an issue.  In fact, Microsoft states that IOPS requirements for Cached Exchange Mode and Online mode are essentially equal now, meaning there is no I/O “penalty” for using Online mode.

Exchange 2010 Performance

Let’s take an example scenario of 500 VDI users all running Outlook in Online mode against an Exchange 2010 backend.  We’ll estimate high and assume they all have a mailbox profile of 300 messages sent/received per day.  According to Microsoft that amounts to a 0.3 IOPS per user requirement of 150 IOPS total, or roughly equivalent to the capabilities of one 15k RPM disk.

We can make it worse and assume that all 500 users also have Blackberry devices, which introduces a multiplier of 2.16 IOPS per user.  (Note: this number is specific to Exchange 2007 as I haven’t been able to find a definitive number for Exchange 2010, but expect that the multipler will be even lower)  That brings our IOPS per user calculation to 0.65 (2.16 x 0.3), bringing the total IOPS requirement for all 500 users to 324 IOPS or roughly three 15K RPM disks (excluding RAID penalty, though this is less of an issue on modern storage arrays).  It is unlikely that all 500 users would have Blackberry devices and also send/receive 300 messages/day, so the actual requirement is likely lower than that.

Searching in Outlook

What about searching from Outlook?  Doesn’t that impose a significant IOPS burden on the Exchange server?  According to Microsoft, the penalty for searching in Online mode is just 10-15% of the database I/O based on user profile.  Using our 300 messages sent/receive per day number of 0.3 IOPS/user, 15% of 0.3 is just 0.045 additional IOPS. It’s no wonder that Microsoft states “Search catalog read I/O occurs when clients issue search queries, and it’s a rare enough occurrence to not be relevant to Exchange 2010 storage design.”

In other words – it doesn’t impact the Exchange environment enough to matter.  Is searching in Online mode slower than using Outlook Instant Search?  No question it’s slower, but search performance has improved significantly with Exchange 2010 so it isn’t as bad as it was in the past.  And the impact on the performance of the Exchange environment is almost negligible.

Summary

Quick breakdown of the numbers used above:

500 users each with a Blackberry and sending/receiving 300 messages/day = 324 IOPS

Impact of all 500 users performing searches = 0.3 x 15% x 500 = 22.5 additional IOPS

Drawbacks to Online Mode

It’s not all rainbows and unicorns when it comes to using Outlook in Online mode unfortunately.  In my opinion the major drawback of using Outlook in Online Mode is not performance but rather availability.

In Exchange 2010, the RPC endpoint has been moved from the Mailbox server to the Client Access Server.  This change means, among other things, that users are not interrupted when moving databases between nodes in a DAG.  Unfortunately that is only true if the user is using Cached Exchange Mode, as Online Mode users experience a brief period where Outlook becomes frozen and unresponsive.

Similarly, if the Exchange environment experiences an outage then Online Mode users are completely frozen and locked out of their mailbox with no access to messages.  Cached Exchange Mode users would not be able to send and receive new messages but would still have access to the contents of their mailbox since they are working off a locally cached copy.

Conclusion

There are many significant advances in Exchange 2010 that have reduced the overall disk I/O requirements down to a tiny fraction of what was required in previous versions.  There is no longer any penalty for using Online Mode vs. Cached Exchange Mode in terms of IOPS required per user, and the example above clearly shows that it is easy to meet the I/O demands of even heavy Exchange users.  The user experience may be slightly better with Cached Exchange Mode but likely not by much.

Most of the issues with Outlook in a VDI environment are around availability, not performance.  Although these cannot be eliminated, properly architecting your Exchange 2010 environment can help eliminate single points of failure and provide excellent availability to users.

If you’re considering VDI and are concerned about Outlook performance, I’d strongly recommend moving to Exchange 2010.  Many of the problems are addressed in Exchange 2010 and it can deliver a good Outlook experience for all VDI users.

The basic premise of the article is this – you can achieve high availability with Exchange 2010 without the use of Database Availability Groups (DAG) by using a combination of VMware HA and Symantec’s ApplicationHA.  Application HA leverages APIs exposed in vSphere 4.1 that let third party applications interact and communicate with VMware HA to perform various functions such as restarting the guest if the entire application has failed.  ApplicationHA can also be used to restart individual guest services, so it can provide a level of application awareness beyond what VMware HA can do alone.

Joe and I agree that the blog is technically correct in that you can get high availability for Exchange 2010 without using DAGs if you use these tools.  Where we don’t agree with the author is in the usefulness of this kind of solution compared to the functionality that a DAG actually provides.

More than just HA

DAGs provide high availability of the Exchange environment but they do much more than that.  They facilitate DR scenarios by including a native replication technology that lets organizations easily replicate their email data to another site that can be brought online quickly and safely.  You can also easily move databases between Mailbox servers within the same site for routine system maintenance, Windows patches, etc.  DAGs can also detect a failure and activate another database copy quickly, most times (but not always) quicker than a VM can reboot.  Additionally, because a reboot can be required with an ApplicationHA/VMware HA solution, there is technically an outage to the end users and the end user experience associated with a reboot repopulating the Exchange database cache isn’t ideal.

DAGs provide high availability at the storage layer

Since a DAG allows up to 16 copies of any mailbox database, it can provide high availability at the storage layer in addition to just protecting the server and Exchange services.  This allows you to put your databases on completely separate storage platforms for a higher degree of availability and protection from many kinds of storage-related failures.  Since Exchange 2010 enables and encourages larger mailboxes and databases, recovery time in the event of catastrophic server failure can be significant if some kind of storage high availability or resiliency isn’t in place.

Management separation

In many firms, the engineering team responsible for Exchange may not be the same as the team responsible for the virtual environment.  In these cases, it is important to understand that leveraging a DAG would allow the Exchange team to be wholly responsible for high availability of the Exchange application.  Leveraging ApplicationHA/VMware HA to protect Exchange would require coordination between the Exchange and virtualization teams for any high availability issues/troubleshooting and that may not be appropriate or optimal in many environments.

Microsoft supported technology

Whenever you go down the road of protecting Microsoft applications with tools that are provided by third party vendors, you run the risk of having issues with Microsoft support.  This used to be more of an issue in the earlier days when host based replication tools were used to protect Exchange, SQL, etc., and is likely less of a problem these days.  I suspect that Symantec has done the right thing here and ApplicationHA does not cause issues with Microsoft support, but having never used it I can’t say for sure.

Cost

When it comes to protecting Tier-1 applications like Exchange cost is important but is not often the driving factor in making a technology decision.  Email has become one of the most important applications in the organization so firms are more willing to pay up to protect it.  That said, the cost of Symantec’s ApplicationHA seems high compared to the costs of implementing a DAG.  With a DAG, you’re just paying for an extra Windows license (which may not have a capital cost depending on your Windows licensing model) and an additional Exchange license.  That may come out cheaper than ApplicationHA, which is licensed in packs of 5 protected servers.  Again cost is usually not the most important factor, but paying less and getting more functionality out of a DAG makes it an attractive option.

Final thoughts

I strongly believe that vSphere provides a great platform for virtualizing Tier-1 applications like Exchange.  Joe and I have worked with clients that have virtualized Exchange 2010 for thousands of users on top of vSphere with great success.  Yet we both feel that the value that DAGs add to Exchange 2010 makes using them on your virtualized Exchange 2010 deployment well worth the added planning, storage requirements, and restrictions.

In my opinion there are some areas where using ApplicationHA and VMware HA guest level monitoring might be useful, such as:

• Small environments where there is no budget for additional storage purchases to accommodate multiple database copies.
• There is a requirement for using vMotion on Mailbox servers (which is not supported in combination with DAGs).  I personally cannot think of a use case where I would want to use vMotion over moving databases between nodes in a DAG, but I recognize there may be situations where this may be desirable.
• An organization already has an investment in a host based or storage based replication product for Exchange but still wants server high availability.  Again, with the native replication functionality of Exchange 2010 (and the 3rd party replication API for the DAG) there are less reasons to use features other than what the DAG provides natively.

I believe that ApplicationHA and using VMware HA for guest level monitoring is a huge step in the right direction.  There are many other use cases for this technology and I think we’ll start to see more people enabling this for nearly all workloads, particularly the native VM monitoring provided by VMware HA.  In the case of Exchange 2010, I think the benefits of DAGs outweigh the benefits that using ApplicationHA provide.

I am by no means an analyst, but to me this shows VMware’s taking desktop virtualization seriously by differentiating it from its core products and making a dedicated certification track.  Citrix has done this for a while with their Citrix Certified Administrator certifications for XenDesktop as they recognized the importance of differentiating that technology from their other offerings.  VMware’s three level desktop certification track shows VMware recognizes they need to build the skill set and acknowledge those that have developed an expertise in delivering their VDI solution.

I also think this brings up an important point about desktop virtualization that I’ve seen as a larger trend in the industry.  That is, the folks that know how to do server virtualization are not automatically the same folks who should be doing desktop virtualization.  Just because someone has a VCP certification doesn’t mean they know how to deploy VMware View (or XenDesktop for that matter).  Desktop virtualization requires a different skillset and creating certifications specifically for this technology shows VMware acknowledges this as well.

I think VMware has made the right move here in creating a seperate certification track for those that focus on desktop virtualization.  It should help both acknowledge those that are experts in desktop virtualization as well as improve the overall design, implementation, and delivery of their desktop virtualization products.

Unidesk offers a smart approach to managing virtual desktops – they break down each component into a layer and then allow you to update and manage each layer separately.  One of the great benefits of VDI in general is the ability to separate the operating system from the applications and from the user’s personality, but that typically involves different tools or different processes to update each component.  Unidesk’s key benefit (in my opinion) is that all of that is now centrally managed in one location and updated through one set of tools.

Law firms are unique in the sheer amount of applications that are run on a typical user desktop and many of those applications are integrated together.  Using tools like application virtualization can help, but historically integrated applications can present a challenge.  Unidesk’s layers show some promise here and there are already success stories from law firms testing the software.

I only had a short time to play with the software but liked what I saw.  I like that the user personality is considered another layer and can be managed without using approaches like roaming profiles or profile management software (which may add cost).  I really like that this product may allow the use of a cheaper version of VMware View or Citrix XenDesktop as some of the more advanced features can be managed (arguably better) by Unidesk.  And finally I really like that the Chief Solution Architect is Ron Oglesby, a veteran of the virtualization community for many years.  He gives the company and the product instant credibility.

I’m looking forward to seeing what comes next from Unidesk.

In VMware vSphere, there are several native Path Selection Policies (PSP) that handle how the ESX or ESXi hosts connect to the storage infrastructure.  For best performance, most use VMware’s native for Round Robin PSP for iSCSI MPIO.  This allows you to better utilize all of your NICs rather than keeping the paths in an active/standby configuration.  In addition to the native policies, VMware has also opened this up to storage vendors to write their own PSPs to take better advantage of their storage arrays.

Previously only available in vSphere Enterprise Plus, with the release of vSphere 4.1 VMware has reduced the licensing requirement to use vendor supplied PSPs to the Enterprise level.  At the same time, Dell released their own PSP for best performance when using EqualLogic SANs.

So why does it make sense to use the MEM and specifically if you’re using Windows Failover Clusters?  Here are a few reasons:

1) VMware does not support virtualized Failover Clusters using the native Round Robin policy.  They also technically do not support Microsoft clusters on iSCSI storage either, but VMware has stated that that restriction will be going away soon.  See this link for more info on virtualizing Microsoft clusters (PDF link).

2) Dell has shown improved performance when using the MEM on virtualized clusters, particularly with Microsoft Exchange 2010 VMs.

3) Installation and configuration of the MEM is very easy and can be configured on a per VMFS volume level.  This would allow you to only use the MEM for the VMFS partitions where your clustered VMs reside.  That said, the benefits of the MEM extend to non-clustered VMs and should probably be used for all VMFS partitions.

Today I decided to take that concept and extend it to systems that you might manage alongside your VI3/vSphere environment.  Storage management seemed like the obvious first choice.

I created an XML file called EqualLogic.xml in C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Plugins\SAN Management.  The contents of the file are as follows (you would replace the <url> section with the IP or DNS name of your SAN):

<scriptConfiguration version=”1.0.0″>
<key>EqualLogic</key>
<description>EqualLogic SAN Management</description>
<view parent=”Inventory.HostSystem”>
<title locale=”en”>EqualLogic</title>
<url>http://10.1.97.30</url>
</view>
</scriptConfiguration>

This gave me a nice way to manage my SAN from the same interface that I use to manage my vSphere environment.  It is simply opening a browser window within the vSphere Client and letting me manage the SAN.

The code above will make the EqualLogic tab visible only when clicking on an ESX/ESXi host.  If you wanted to extend that to other objects, you can simply adjust the “<view parent=” section.  For example, to also make this available at the cluster level you would include the following:

<view parent=”Inventory.Cluster”>
<title locale=”en”>EqualLogic</title>
<url>http://10.1.97.30/</url>
</view>

Similarly you could add Inventory.Global, Inventory.VirtualMachine, Inventory.Datacenter, etc.

This is a really simple way to make it easy to manage any web interface (not just EqualLogic) from within the vSphere/VI Client.  It’s not a new trick and has been out there for a while but I had never used it for this until today.

In my recent post on the effects of ASLR in vSphere the comments turned into a discussion about TPS on modern processors. And there are countless posts about this issue on the VMTN forums where folks are looking for a fix. In reality nothing is broken and there is no need to fix the issue.

VMware has published a KB article that gives more information on TPS with Nehalem processors and why it appears TPS isn’t working (this affects modern AMD processors also). The short version is that TPS uses small pages (4K), and Nehalem processors utilize large pages (2MB). The ESX/ESXi host keeps track of what pages could be shared, and once memory is over-comitted it breaks the large pages into small pages and begins sharing memory.

Many people think this is a bug in ESX that needs to be fixed. This likely started because when vSphere 4 was released there was a bug around memory usage on ESX hosts with Nehalem processors. In reality the bug was that vCenter was triggering high memory usage alarms for virtual machines running in this configuration. Nothing was actually wrong but because the host was using all of the assigned memory for the VM, vCenter was incorrectly triggering the alarm. That behavior has since been fixed with a patch and is no longer an issue.

So what does this actually look like? When a VM is powered up on an ESX host with Nehalem processors, the amount of host memory in use will not drop down as the VM uses less memory or becomes idle. Those of us that have been using ESX for a long time likely found this scenario disturbing.

From vSphere Client (red highlighted section shows guest taking all of the 2GB assigned memory, yet memory usage in the guest is very low):

From esxtop (red highlighted section shows almost no memory being shared with page sharing):

The above screenshots show a host that is under-committed on memory and so no page sharing is occurring.  If the host gets over-commited page sharing kicks in automatically by breaking up large pages into small pages.  You can force the use of small pages on all guests all the time by changing the value of the advanced option Mem.AllocGuestLargePage to 0.  I don’t really see any reason to do this – remember that TPS isn’t broken and what you see in the above screenshots is normal and expected.

Once host memory is over-committed (or if you use the advanced option), memory sharing kicks in and things look like they normally do when page sharing is taking place.

From the vSphere Client (red highlighted section shows guest taking very little of the assigned 2GB memory as page sharing has kicked in):

From esxtop (red highlighted sections show a large amount of shared memory and the host is over-commited on memory by 48%):

A quick note on the esxtop screenshot above – it was taken from a VDI environment where all workloads are identical so that explains the high amount of shared pages.  It was also overcommitted more than normal as it was taken during host maintenance.

I hope this clears up some of the confusion around TPS on modern Intel/AMD processors.  In short, don’t get hung up on the fact that TPS isn’t kicking in like it did with older processors.  Nothing is broken, TPS is working as expected, and it will kick in when you actually need it.

I’m working on a vSphere implementation using Dell EqualLogic SANs and wanted to configure Round Robin on all of my datastores. Dell has a great whitepaper on how to set this up, but unfortunately the document fails to mention one key thing: this doesn’t change the default path selection plugin (PSP) from Fixed to Round Robin.   That means that you’ll have to set the multipathing policy to Round Robin on all of your existing datastores and will have to remember to do that on all future datastores. When you’ve got multiple ESX hosts with lots of  datastores this can quickly become a pain.

Luckily there is a way to force the default multipathing policy to Round Robin. The following commands can be used to change the default PSP to Round Robin as well as configure round robin specifically for the EqualLogic provider.  These commands can be entered at the Service Console or via the vSphere CLI 4.0:

esxcli nmp satp setdefaultpsp –satp VMW_SATP_DEFAULT_AA –psp VMW_PSP_RR
esxcli nmp satp setdefaultpsp –satp VMW_SATP_EQL –psp VMW_PSP_RR
esxcli corestorage claimrule load
esxcli corestorage claimrule run

Note that “satp” and “psp” are preceded by two dashes and not a single dash as it appears in this blog post.

Once you enter those commands (no rebooting required) any volume you add, either new or existing, will use Round Robin MPIO by default.

Address Space Layout Randomization (ASLR) is a security feature that randomizes the position of data in memory, making it more difficult for attackers to predict where data can be found while in memory. This feature has been enabled in Windows since Windows Vista, and other operating system such as Linux and MacOS implement this in some form as well.

Since ASLR randomizes information in memory it makes sense that it would be more difficult for TPS to find identical memory pages and thus memory sharing would be reduced. But just how much of a difference does it make? I decided to try and find out. Here are the specs from my test environment:

Server: HP DL385 G1 (AMD Opteron 275)
ESX: 4.0.0 build 244038
Guest OS: Windows Server 2008 R2
Guest RAM: 2.5GB

All guests were cloned from the same template and have the same software installed. On guests TESTSRV1 and TESTSRV3, I left the default settings. On TESTSRV2 and TESTSRV4, I disabled ASLR using the following regkey:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management]
“MoveImages”=dword:00000000

In all of my testing, including leaving VMs idle and also running memory tests, I found no measurable difference in the amount of memory shared with TPS. I also looked at esxtop to see how much memory was actually being shared and I saw virtually no difference whatsoever between VMs that had ASLR enabled and those that had it disabled.

Host Memory Usage:

esxtop statistics:

The SHRD and SHRDSVD columns represent how much memory is being shared with TPS and the total memory savings. Clearly there is quite a bit of memory sharing going with or without ASLR enabled.

Why would this be the case since it makes sense that TPS would be hurt by ASLR? ASLR requires applications to “opt-in” to have their memory randomized, and I suspect that much of Windows 2008 R2 is not opted in. Perhaps applications will come out in the future that are written to take advantage of ASLR, but at the moment that doesn’t appear to be the case.

Of course this is by no means a definitive test as it wasn’t run with production systems and real users running real applications. That said, I think it shows that ASLR does not dramatically reduce the amount of memory shared with TPS. I did also look at production systems left at the default settings (ASLR enabled) and saw similar memory sharing gains. I’m curious if others have seen similar results in their environments, so drop me a line if you’ve done any similar testing.

More info:

What is ASLR (Wikipedia)
Interpreting esxtop statistics

When virtualization became popular it presented a challenge to those looking to continue to use their tape drives in fully virtualized environments. If you were using VMware you could use SCSI pass-through to present a tape drive or changer directly to a virtual machine but that prevented you from using any advanced features like VMotion. It also tied your tape drive and VM to a single host containing a SCSI card, making things complicated if that host were to experience a hardware failure.

While this is still possible in vSphere 4 (and the previous version), this configuration is not ideal.  Instead, consider converting that SCSI tape changer into an iSCSI target that can be used on any virtual machine attached to any host by using an iSCSI-to-SCSI bridge.  These bridges let you attach your tape changer directly to the device and then present the tape changer to virtual machines as an iSCSI target.  There are several different vendors providing this technology, including Atto Technology, Paralan, and others.

Once the tape drive is attached to the iSCSI bridge and configured as a target, you simply use the Microsoft iSCSI initiator inside a virtual machine to connect to the device.  The tape device will appear to the virtual machine as if it were any other iSCSI target (like a SAN LUN).

After connecting to the target in the iSCSI Initator, the tape device will become visible in Device Manager on the VM and within tape backup software.

After the tape device has been successfully discovered, the virtual machine can then be managed with features like VMotion, HA, and DRS because the VM won’t be tied to an individual host.  This configuration also opens up other design possibilities, such as multiple backup servers running different backup products.  Using the iSCSI bridge provides a lot more flexibility than directly attaching the tape device to an ESX host.

Sounds great, right?  As always, there are things to consider before moving forward with this type of solution:

1) Does your backup software vendor and tape changer vendor support this setup?

2) Will this setup meet the performance requirements of your environment?  In practice I’ve seen these devices push 2 GB/min or more, similar to the performance of direct attached tape devices.

Using an iSCSI-to-SCSI bridge opens up a lot of possibilities for keeping a tape device in your fully virtualized environment.  It also simplifies your setup and allows you to take advantage of enterprise features of your virtualization product.  Finally, for around $1,200 for the iSCSI bridge this is also an affordable solution to a common problem.