(As an aside, this post is only talking to VDI and in particular as a desktop replacement and not other use cases that may include, but are not limited to training rooms, war rooms, remote access, etc.)

Licensing is confusing and expensive

The finger is first pointed at Microsoft and their ever changing (confusing) licensing scheme for VDI.   A quick browse of the VDI licensing guide details seven scenarios and the licensing implications for using VDI with respect to each.  Does it really need to be this confusing?  Can’t Microsoft license VDI in the same way they license Terminal Servers to simplify this?  The other hindrance here is the essential requirement that an organization purchase Software Assurance for desktops or purchase a VDA for each user/device connecting to a VDI desktop.  This is a cost of about $100 per user/desktop that some firms have not been used to shelling out.  Most of the firms that I have worked with (specifically the SMBs) don’t purchase Software Assurance for Windows desktops as they purchase Windows OEM licenses from Dell, HP, etc.

Let’s now point the finger at Citrix.  I have a law firm client at the moment that currently leverages XenApp as a desktop replacement and owns a XenApp license for every user in the firm.  “Upgrading” these licenses to XenDesktop using Citrix’s trade up calculator results in the firm paying $20,000 (retail) to convert their existing (140) concurrent use XenApp licenses to named user/device XenDesktop licenses.  Should they want to continue to leverage CCU licenses for XenDesktop, they would have to shell out almost $50,000….  ‘Yes’, they are technically now getting XenDesktop and XenApp as part of the upgrade, but I would contend that if Citrix allowed a 1:1 conversion from XenApp CCU licenses to XenDesktop  CCU licenses (and did not give XenApp for “free” as part of it), it would be more palpable for customers (but clearly less profitable for Citrix) to make the conversion.

Citrix and VMware’s Offline VDI solutions are not there yet

Let me start by acknowledging there are less and less scenarios in which a user is “offline” the days.  However there are still scenarios in which users are offline or have a degraded connection that require a functional offline VDI solution.  Specifically, airplanes, spotty WiFi in hotels (my personal pet peeve) and public spots along with low bandwidth/high latency air cards among other scenarios.  VMware’s botched approach with its Type II hypervisor with checking in/out VMs would not meet expectations of law firm users.  Citrix’s approach with its Type I hypervisor, XenClient, is promising but is still a 1.0 product one year later.  The HCL for XenClient is also limited to a subset of machines and the server piece (Synchronizer) of XenClient is also only available for Citrix XenServer, which also makes it a non starter.  The bottom line is that the current offerings by Citrix and VMware are just not good enough from a usability/manageability perspective for law firms to accept it.

Organization politics and delegation of responsibilities

This section is very specific to the legal space as most firms are structured such that there are “Infrastructure” folks and “Applications” folks that work in individual silos.  The Infrastructure folks are primarily responsible for networking, security, messaging, virtualization, shared storage, sometimes Citrix XenApp, etc. while the “Applications” folks are responsible for Windows desktops, user profile management, software distribution, application integration and the overall user experience.  My experience has been that most Applications folks don’t want to touch the Infrastructure, and Infrastructure folks definitely don’t want to deal with anything user facing.  If a firm were to embark on a VDI initiative, it would require tight collaboration between the two different silos as a successful VDI implementation requires careful infrastructure planning as well as application integration for good user experience.  Of course, fighting organizational politics and contention of responsibilities is left up to a CIO, but I see this is another reason why VDI goes nowhere in law firms.

User Experience is not as good as local desktops

Isn’t this what a desktop is all about?  If a firm invests hundreds of thousands of dollars in a new VDI desktop, and the user perception is that it doesn’t perform as well as the traditional desktop it was replacing, is the project a success?  PCoIP and ICA protocols have improved significantly along with some of the HDX and user experience improvement with the latest versions of XenDesktop and View, however designing a VDI desktop to perform as well as a typical local desktop (4 cores/4 GB RAM) with Windows 7 is still difficult to impossible.  Specifically when it comes to media (audio and video) and resource intensive applications.

The local desktop is still required

Ideally, a VDI environment would leverage a BYOPC or Thin Terminal model in which the local corporate desktop is obsolete and not required.  Users would have their own PC and Thin Terminals to access their “managed” corporate VDI desktop in which all their applications were available to them.  Unfortunately, its not that black and white at law firms.  To start, law firms have hundreds of applications and making them all available through a VDI desktop is very challenging even with application virtualization solutions.  Additionally, the local desktop in some way provides an “insurance policy” for firms to allow for local applications (outside the VDI desktop) in some facet.  This could be business applications such as a browser toolbar required for collaboration, or  Adobe PDF reader to view large PDF files a user brings in on a DVD.  These local applications could also be non-business applications such as DVD video software and iTunes.  The point is that law firms have knowledge workers who have an array of IT demands with respect to their applications and user experience and the local desktop would not get replaced
even with VDI.  I am not saying it is impossible, it would just be very hard.

The fact that the local desktop is still required creates multiple management points for IT as they have to not only manage a VDI desktop, they have to continue to maintain, service, patch, etc. a local Windows desktop.  Citrix and VMware would contend that the capital costs of VDI is not where the savings are, but the operational costs are where all the savings come from.  I am sure there are some fancy marketing charts or Gartner charts that attempt to prove this, but let’s take them at their word.  The challenge here is that most law firm’s would not be replacing their local desktops (with Thin Terminals) should they move forward with a VDI solution.

Honorable Mention

Capital costs for shared storage and virtualization infrastructure, Thin environments with XenApp/Terminal Servers can be built for a fraction of the cost of VDI.

Conclusions

I am going to end this post by saying the success to any VDI environment (or any technology project for that matter) is knowing your use case and business requirements.  This has been drilled into me by my colleague Matt Liebowitz who probably starts every VDI conversation with the phrase “know your use case”.  Too many times technologists lose sight of the business requirements and use cases for a particular technology before attempting to fit a square peg in a round hole.  VDI offers obvious benefits in some areas over a traditional desktop, but it isn’t a one size fits all model.   The use case I focus on in this post (VDI as a desktop replacements) relay my thoughts on why most firms see VDI as an non starter because the challenges associated with implementing VDI outweighs the firm’s business requirements for doing it.  Is VDI a non starter for all law firms?  Absolutely not. There are some scenarios in which a firm’s business requirements and existing desktop circumstances make VDI an attractive and potentially more cost effective approach than a traditional desktop.

Typically I’ve seen firms start small and grow the functions leveraging their NetScalers over time.  They might replace their Secure Gateways with virtual Access Gateways running on the VPX with the idea to add Exchange 2010 traffic to it down the road.

If you find yourself in a similar situation you can use the chart below to size the needed VPX based on some of the most common traffic types.  Also note that it’s very easy to purchase a small model and upgrade later should you have bandwidth requirement changes.

For a copy of the quick-and-dirty Excel spreadsheet I’ve used to calculate bandwidth needs feel free to email me: Clinton Gates

As I investigated the performance issues I reviewed quite a bit of product documentation, best practice guides, forum posts, etc. and spent quite a bit of time on the phone with Citrix technical support. Still, the issues remained. I then attended a VDI summit at Microsoft and was fortunate enough to speak to a Citrix engineer who happened to be presenting. As we took a walk down the hall I described the issues and environment, and by the time we got back he’d identified the problem and the solution. Though at first blush locating the vDisk and write cache on a highly available network share appears to make sense, doing so eliminates PvS’s use of the Windows system cache.

The system cache stores frequently accessed information in RAM but only applies to local files (or files that appear local, as is the case with an iSCSI drive). So in the current environment every time a target VM issued a read request it was sent through the network to a PvS, through the network again to the active file cluster node, through the network again to the SAN, then back to the file cluster node, back to the PvS, and back to the target VM. Granted, approximately 50% of the traffic “through the network” was through a virtual switch since each target VM had a 50% chance of utilizing the PvS on the same host as the active file cluster node, but the point remains – each request resulted in some degree of physical network traffic, OS overhead on both the PvS and file cluster node, and in read I/O on the SAN.

The solution was to create local storage on each PvS, relocate the vDisk from the file cluster to the local storage (now two sets of files instead of one), modifying the vDisk Store properties to point to the local storage, and defining a process to keep vDisk files in sync between the PvSs as updates are made. Also, most importantly, I increased the RAM on each PvS from 2Gb to 6Gb. Together, these changes allowed the most frequently accessed information from the vDisk to be cached in RAM on each PvS, thus reducing read IOPS, read latency, and write latency as shown below, and eliminating the trip through the file cluster node.

Note that read IOPS decreased 27% despite a 16% increase overall. Further analysis indicates the actual impact was higher:

Stated another way, if total IOPS were constant at 998.95 from week 1 to week 2, based on the numbers above we would expect read IOPS to be 22% of that in week 2. That’s 217.92, which is 37% less than 346.40.

It’s important to note that with approximately 25 non-VDI servers utilizing the same storage infrastructure these are certainly not isolated results – the statistics above cannot be attributed to the vDisk relocation alone. However, since the environment was fairly static for the periods immediately preceding and following this project it’s assumed that 1) non-VDI activity remained roughly constant, and 2) the 16% increase is due to the movement of several sets of vDisk files within the SAN. Obviously every environment is different in terms of equipment, topology, workload, etc. Therefore this analysis is intended to illustrate the performance gains in one particular environment, not to state that you will have the same gains in your environment.

In any event, the changes described above immediately improved VDI performance, and a few weeks later relocating write cache from the file cluster to a local disk on each VM improved performance further. Specifically, relocating the write cache left write-related SAN I/O roughly constant but eliminated the network traffic and OS overhead associated with PvS and the file cluster node. This also allowed us to decommission the file cluster nodes and reclaim the host and SAN resources they consumed. It’s worth noting that since the VDI environment shares the SAN with the production servers, the decreased read IOPS and corresponding decreases in both read and write latency provided benefits system-wide.

I found this solution interesting from a technical perspective, but equally interesting that so little information is out there on the impact of vDisk location. The Citrix support staff I spent hours on the phone apparently weren’t aware of it, and Citrix’ “Implementing a Virtual Desktop Infrastructure” course didn’t cover it (though ironically the course materials show a vDisk store with a UNC path). Also, the product documentation doesn’t cover it and the best practices guide arguably contains misleading information:

All traffic that occurs between the vDisk and the target device passes through the PVS machine regardless of where the vDisk resides. Using Windows Server 2003 file caching features can improve vDisk deployment efficiency.

A fair read is that “all” vDisk traffic “can be improved” by caching, when in fact “only” traffic to a local vDisk is affected, and it “will dramatically be improved.”

Now that I know what I’m looking for, I’ve found recommendations relative to vDisk location and system cache in a blog post, a forum post, and a handful of similar resources elsewhere, but given the significant performance impact to both VDI and other components that utilize the same storage infrastructure I’m surprised that this topic isn’t discussed more prominently in more formal documentation. That said, I hope this information is helpful to anyone planning or troubleshooting a similar environment.

I recommend reading Andre’s post to get more information on this topic.  The short version is this – Exchange/Outlook best practices do not necessarily work in a VDI environment.

When using Outlook in an Exchange environment, it is recommended to use Cached Exchange Mode.  In this mode a copy of the user’s mailbox is downloaded into an OST file and stored offline on the user’s desktop.  This mode offers better performance for the end user and reduces utilization on the Exchange environment as well.  In addition, using Cached Exchange Mode allows users to use Outlook Instant Search for fast searching of items in their mailbox.  Instant Search works by indexing the contents of the OST file so all searches occur locally and not on the Exchange server, further improving performance and reducing utilization on Exchange.

VDI environments that we see at our clients are typically configured as non-persistent or floating pools of desktops.  That is, each user connects to a pool of identical desktops and grabs whatever desktop is available.  When the user logs off, any changes written to the VDI desktop are discarded and the desktop returns to a pristine state.  There are mechanisms and tools in place to make sure user data is retained at logoff.

So if user data is retained at logoff, why can’t we use Cached Exchange Mode in non-persistent VDI environments?

• The OST file is equal in size to the user’s mailbox so storing a 15-30GB OST (not unusual at our clients) is not that practical from a performance or storage perspective.  If this data is being stored on the SAN, then you’re essentially doubling your Exchange storage (which may already be doubled or tripled if you’re using Exchange 2010 w/ DAGs).  In addition, the length of time it would take to download that file every time and the I/O impact that would cause makes it completely impractical.
• OST files are not supported when stored on network shares, so redirecting the OST to a home directory is out.
• Indexing of files on virtual desktops is typically disabled to reduce I/O demands.  This would prevent the use of Outlook Instant Search even if the OST was present.

For these and other reasons, Outlook is typically configured in Online mode when used with VDI.  This keeps all mailbox operations and searches on the Exchange server, placing the processing and I/O burden solely on the Exchange environment.  That sounds bad, but advances in Exchange technology specifically with Exchange 2010 have made this much less of an issue.  In fact, Microsoft states that IOPS requirements for Cached Exchange Mode and Online mode are essentially equal now, meaning there is no I/O “penalty” for using Online mode.

Exchange 2010 Performance

Let’s take an example scenario of 500 VDI users all running Outlook in Online mode against an Exchange 2010 backend.  We’ll estimate high and assume they all have a mailbox profile of 300 messages sent/received per day.  According to Microsoft that amounts to a 0.3 IOPS per user requirement of 150 IOPS total, or roughly equivalent to the capabilities of one 15k RPM disk.

We can make it worse and assume that all 500 users also have Blackberry devices, which introduces a multiplier of 2.16 IOPS per user.  (Note: this number is specific to Exchange 2007 as I haven’t been able to find a definitive number for Exchange 2010, but expect that the multipler will be even lower)  That brings our IOPS per user calculation to 0.65 (2.16 x 0.3), bringing the total IOPS requirement for all 500 users to 324 IOPS or roughly three 15K RPM disks (excluding RAID penalty, though this is less of an issue on modern storage arrays).  It is unlikely that all 500 users would have Blackberry devices and also send/receive 300 messages/day, so the actual requirement is likely lower than that.

Searching in Outlook

What about searching from Outlook?  Doesn’t that impose a significant IOPS burden on the Exchange server?  According to Microsoft, the penalty for searching in Online mode is just 10-15% of the database I/O based on user profile.  Using our 300 messages sent/receive per day number of 0.3 IOPS/user, 15% of 0.3 is just 0.045 additional IOPS. It’s no wonder that Microsoft states “Search catalog read I/O occurs when clients issue search queries, and it’s a rare enough occurrence to not be relevant to Exchange 2010 storage design.”

In other words – it doesn’t impact the Exchange environment enough to matter.  Is searching in Online mode slower than using Outlook Instant Search?  No question it’s slower, but search performance has improved significantly with Exchange 2010 so it isn’t as bad as it was in the past.  And the impact on the performance of the Exchange environment is almost negligible.

Summary

Quick breakdown of the numbers used above:

500 users each with a Blackberry and sending/receiving 300 messages/day = 324 IOPS

Impact of all 500 users performing searches = 0.3 x 15% x 500 = 22.5 additional IOPS

Drawbacks to Online Mode

It’s not all rainbows and unicorns when it comes to using Outlook in Online mode unfortunately.  In my opinion the major drawback of using Outlook in Online Mode is not performance but rather availability.

In Exchange 2010, the RPC endpoint has been moved from the Mailbox server to the Client Access Server.  This change means, among other things, that users are not interrupted when moving databases between nodes in a DAG.  Unfortunately that is only true if the user is using Cached Exchange Mode, as Online Mode users experience a brief period where Outlook becomes frozen and unresponsive.

Similarly, if the Exchange environment experiences an outage then Online Mode users are completely frozen and locked out of their mailbox with no access to messages.  Cached Exchange Mode users would not be able to send and receive new messages but would still have access to the contents of their mailbox since they are working off a locally cached copy.

Conclusion

There are many significant advances in Exchange 2010 that have reduced the overall disk I/O requirements down to a tiny fraction of what was required in previous versions.  There is no longer any penalty for using Online Mode vs. Cached Exchange Mode in terms of IOPS required per user, and the example above clearly shows that it is easy to meet the I/O demands of even heavy Exchange users.  The user experience may be slightly better with Cached Exchange Mode but likely not by much.

Most of the issues with Outlook in a VDI environment are around availability, not performance.  Although these cannot be eliminated, properly architecting your Exchange 2010 environment can help eliminate single points of failure and provide excellent availability to users.

If you’re considering VDI and are concerned about Outlook performance, I’d strongly recommend moving to Exchange 2010.  Many of the problems are addressed in Exchange 2010 and it can deliver a good Outlook experience for all VDI users.

The Citrix Certified Enterprise Engineer for Virtualization certification is designed to certify breadth of knowledge across Citrix virtualization solutions. The CCEE offers experienced IT professionals the opportunity to develop and validate their ability to combine the coordination of operational planning efforts with tactical design expertise and integration know-how.

Niraj and Jeff have a well established reputation for helping law firms adopt thin-client technologies and best practices around enterprise desktop and application management. It is exciting to watch server-based computing strategies mature as new virtual desktop solutions, such as VDI and application virtualization, have become more mainstream. Not only are these new tools offering firms technical improvements, but also significant operational efficiencies and flexibility over traditional desktop and server-based approaches.

On behalf of the team, Congratulations to Niraj and Jeff on your accomplishments!

I am by no means an analyst, but to me this shows VMware’s taking desktop virtualization seriously by differentiating it from its core products and making a dedicated certification track.  Citrix has done this for a while with their Citrix Certified Administrator certifications for XenDesktop as they recognized the importance of differentiating that technology from their other offerings.  VMware’s three level desktop certification track shows VMware recognizes they need to build the skill set and acknowledge those that have developed an expertise in delivering their VDI solution.

I also think this brings up an important point about desktop virtualization that I’ve seen as a larger trend in the industry.  That is, the folks that know how to do server virtualization are not automatically the same folks who should be doing desktop virtualization.  Just because someone has a VCP certification doesn’t mean they know how to deploy VMware View (or XenDesktop for that matter).  Desktop virtualization requires a different skillset and creating certifications specifically for this technology shows VMware acknowledges this as well.

I think VMware has made the right move here in creating a seperate certification track for those that focus on desktop virtualization.  It should help both acknowledge those that are experts in desktop virtualization as well as improve the overall design, implementation, and delivery of their desktop virtualization products.

Unidesk offers a smart approach to managing virtual desktops – they break down each component into a layer and then allow you to update and manage each layer separately.  One of the great benefits of VDI in general is the ability to separate the operating system from the applications and from the user’s personality, but that typically involves different tools or different processes to update each component.  Unidesk’s key benefit (in my opinion) is that all of that is now centrally managed in one location and updated through one set of tools.

Law firms are unique in the sheer amount of applications that are run on a typical user desktop and many of those applications are integrated together.  Using tools like application virtualization can help, but historically integrated applications can present a challenge.  Unidesk’s layers show some promise here and there are already success stories from law firms testing the software.

I only had a short time to play with the software but liked what I saw.  I like that the user personality is considered another layer and can be managed without using approaches like roaming profiles or profile management software (which may add cost).  I really like that this product may allow the use of a cheaper version of VMware View or Citrix XenDesktop as some of the more advanced features can be managed (arguably better) by Unidesk.  And finally I really like that the Chief Solution Architect is Ron Oglesby, a veteran of the virtualization community for many years.  He gives the company and the product instant credibility.

I’m looking forward to seeing what comes next from Unidesk.

First and foremost, XenClient has a very strict HCL.  It is specific to the most modern Dell, HP and Lenovo’s business series laptops with Intel Graphics.  This isn’t a surprise and is actually expected to some degree.  I know that Citrix is working closely with OEMs and Intel for compatibility, so this will expand as the product matures and if it finds a niche.  I was working with a test Dell Latitude E4300 as my XenClient laptop and was able to complete the install within 10 minutes.  It was very straightforward and went without a hitch.  The hypervisor takes about a minute to boot up and then you are presented with the main GUI (screenshot below).  The UI is pretty straightforward, and includes functional information like remaining battery life, gives the ability to connect to different wireless networks, and configuring some basic hypervisor’s properties.

Using the wizard driven menu, I created a Windows 7 desktop and that is what was presented at the center of the screen.

After Windows 7 was installed in a VM, I installed the XenTools, but still had an issue with audio playback.  This was resolved easily by installing the Realtek AC97 audio driver.  My test laptop only had 2 GB of RAM on it and although the hypervisor was utilizing 600+ MB of RAM, the 1 GB Windows 7 desktop ran pretty well.  I confirmed the hardware audio buttons worked and functions passed through to the VM and USB devices such as iPhones and aircards were also passed through without error.  Also, because the XenClient supports paravirtualization, it is able to directly pass through the Intel video card to the running VM.  This means Aero support, HD video playback among other functions that require video acceleration.  Check out the screenshot from the running VM.  The Intel GMA graphics driver is passed through into the virtual machine.

Finally, there is the “Synchronizer” piece that is the server backend that the XenClient can connect to.  It allows deploying, updating and managing virtual machine images centrally against connected XenClient hosts.  The Synchronizer Virtual Appliance only supports XenServer at the moment, so it has been a non-starter for me and expect it will be in most other environments as well.  It is rumored that a virtual appliance that supports ESX is slated for release later this year, so that should help make it easier to test.  I’ll do more testing when this is made available.

Anyway, the “1.0” release of XenClient should really be called an extended beta.  The product is very slick, but is far away from any production use.  I would go so far as to say that we are at least 12 months away from XenClient seeing a production environment.  Citrix has a interesting product, but has the challenge of figuring out specifically how XenClient fits into existing XenDesktop and Provisioning Server environments.  Will the Synchronizer be integrated into Provisioning Server in the future?  Will a Provisioning Server vDisk have the ability to be offline with XenClient?  XenClient introduces a lot of questions.

So why do I think is this rumor relevant again?  Last week, Citrix released the Citrix Access Gateway VPX 5.0.  Along with this release came a significant licensing change that cuts into the CSG’s value.  The Access Gateway’s “Platform License” ($995 retail) includes unlimited connections to XenApp/XenDesktop through the Access Gateway.  The “Universal” CALs (Access Gateway licenses) still exist, but now differentiate the VPN, Smart Access other traditional AG features.

So what does this mean?  This means that organizations that only want the SSL proxy functionality of the Access Gateway can do it at it a significantly lower price point than previously.  The price point is lowered because the (expensive) “Universal” licenses are no longer required for SSL XenApp/XenDesktop connections.  If Citrix were to stop supporting or developing the CSG, there is now a viable option from a price point perspective to its existing CSG customers.

Now that we have that squared away, someone would ask the obvious “Why should I pay for the Access Gateway VPX with a platform license, if the CSG is still free?  Free is still better.” – Fair question.  The value is in the new functionality included with the 5.0 platform of the Access Gateway.  The most obvious feature is the new High Availability feature on the 5.0 platform.  AG now supports running in an active/passive configuration on all appliances running 5.0 with the platform license.  This is something the CSG currently does not do and this reason alone is enough to consider moving to an AG 5.0 based appliance.

The other reason is to be forward thinking and understand that although the CSG serves a need today, it is obvious that Citrix is moving towards the AG platform.  While this might mean more of an investment now, the expectation is that it will pay off with newer features and functionality that comes with the AG platform going forward.

XenClient 1.0 will be released later this week for download on MyCitrix and is being demoed and talked about at the conference.  After playing with it at one of the demo stations and talking with some Citrix Engineers, here are some details:

• Unlike VMware’s View, XenClient is a Type 1 hypervisor.  This means it lives above the client side OS (Windows).  Once installed, the user has the option to boot into whatever VMs are available on the laptop.
• At the moment, XenClient will only support a small subset of hardware types.  This includes Dell’s Latitude E series, Dell Optiplex 780, and  HP EliteBook laptops.  Full HCL to be published later in the week.
• Citrix Synchronizer is the server appliance that chats with the XenClient (over HTTP/SSL) to continuously sync and update the local running VM back to the Data Center.
• Synchronizer will be available as a virtual appliance running on XenServer.  According to Citrix, there is no planned version for ESX.  – I am sure this will change though.
• XenClient can be installed as a standalone or in conjunction with “Synchronizer”.
• XenClient supports paravirtualization to allow VMs direct access to hardware (using it’s native driver).  For example, a VM under XenClient can tap directly into a GPU for accelerated video playback and graphic intensive applications.  – This demoed very well with the engineer playing back an HD video file without skipping within a VM.
• At the moment, only a small subset of USB devices are supported through XenClient.

I am at Synergy all week, so I expect to learn more details about XenClient over the course of the week.  If you guys have questions, feel free to post them in the comments and I’ll try to get those questions answered while at the conference.