(As an aside, this post is only talking to VDI and in particular as a desktop replacement and not other use cases that may include, but are not limited to training rooms, war rooms, remote access, etc.)

Licensing is confusing and expensive

The finger is first pointed at Microsoft and their ever changing (confusing) licensing scheme for VDI.   A quick browse of the VDI licensing guide details seven scenarios and the licensing implications for using VDI with respect to each.  Does it really need to be this confusing?  Can’t Microsoft license VDI in the same way they license Terminal Servers to simplify this?  The other hindrance here is the essential requirement that an organization purchase Software Assurance for desktops or purchase a VDA for each user/device connecting to a VDI desktop.  This is a cost of about $100 per user/desktop that some firms have not been used to shelling out.  Most of the firms that I have worked with (specifically the SMBs) don’t purchase Software Assurance for Windows desktops as they purchase Windows OEM licenses from Dell, HP, etc.

Let’s now point the finger at Citrix.  I have a law firm client at the moment that currently leverages XenApp as a desktop replacement and owns a XenApp license for every user in the firm.  “Upgrading” these licenses to XenDesktop using Citrix’s trade up calculator results in the firm paying $20,000 (retail) to convert their existing (140) concurrent use XenApp licenses to named user/device XenDesktop licenses.  Should they want to continue to leverage CCU licenses for XenDesktop, they would have to shell out almost $50,000….  ‘Yes’, they are technically now getting XenDesktop and XenApp as part of the upgrade, but I would contend that if Citrix allowed a 1:1 conversion from XenApp CCU licenses to XenDesktop  CCU licenses (and did not give XenApp for “free” as part of it), it would be more palpable for customers (but clearly less profitable for Citrix) to make the conversion.

Citrix and VMware’s Offline VDI solutions are not there yet

Let me start by acknowledging there are less and less scenarios in which a user is “offline” the days.  However there are still scenarios in which users are offline or have a degraded connection that require a functional offline VDI solution.  Specifically, airplanes, spotty WiFi in hotels (my personal pet peeve) and public spots along with low bandwidth/high latency air cards among other scenarios.  VMware’s botched approach with its Type II hypervisor with checking in/out VMs would not meet expectations of law firm users.  Citrix’s approach with its Type I hypervisor, XenClient, is promising but is still a 1.0 product one year later.  The HCL for XenClient is also limited to a subset of machines and the server piece (Synchronizer) of XenClient is also only available for Citrix XenServer, which also makes it a non starter.  The bottom line is that the current offerings by Citrix and VMware are just not good enough from a usability/manageability perspective for law firms to accept it.

Organization politics and delegation of responsibilities

This section is very specific to the legal space as most firms are structured such that there are “Infrastructure” folks and “Applications” folks that work in individual silos.  The Infrastructure folks are primarily responsible for networking, security, messaging, virtualization, shared storage, sometimes Citrix XenApp, etc. while the “Applications” folks are responsible for Windows desktops, user profile management, software distribution, application integration and the overall user experience.  My experience has been that most Applications folks don’t want to touch the Infrastructure, and Infrastructure folks definitely don’t want to deal with anything user facing.  If a firm were to embark on a VDI initiative, it would require tight collaboration between the two different silos as a successful VDI implementation requires careful infrastructure planning as well as application integration for good user experience.  Of course, fighting organizational politics and contention of responsibilities is left up to a CIO, but I see this is another reason why VDI goes nowhere in law firms.

User Experience is not as good as local desktops

Isn’t this what a desktop is all about?  If a firm invests hundreds of thousands of dollars in a new VDI desktop, and the user perception is that it doesn’t perform as well as the traditional desktop it was replacing, is the project a success?  PCoIP and ICA protocols have improved significantly along with some of the HDX and user experience improvement with the latest versions of XenDesktop and View, however designing a VDI desktop to perform as well as a typical local desktop (4 cores/4 GB RAM) with Windows 7 is still difficult to impossible.  Specifically when it comes to media (audio and video) and resource intensive applications.

The local desktop is still required

Ideally, a VDI environment would leverage a BYOPC or Thin Terminal model in which the local corporate desktop is obsolete and not required.  Users would have their own PC and Thin Terminals to access their “managed” corporate VDI desktop in which all their applications were available to them.  Unfortunately, its not that black and white at law firms.  To start, law firms have hundreds of applications and making them all available through a VDI desktop is very challenging even with application virtualization solutions.  Additionally, the local desktop in some way provides an “insurance policy” for firms to allow for local applications (outside the VDI desktop) in some facet.  This could be business applications such as a browser toolbar required for collaboration, or  Adobe PDF reader to view large PDF files a user brings in on a DVD.  These local applications could also be non-business applications such as DVD video software and iTunes.  The point is that law firms have knowledge workers who have an array of IT demands with respect to their applications and user experience and the local desktop would not get replaced
even with VDI.  I am not saying it is impossible, it would just be very hard.

The fact that the local desktop is still required creates multiple management points for IT as they have to not only manage a VDI desktop, they have to continue to maintain, service, patch, etc. a local Windows desktop.  Citrix and VMware would contend that the capital costs of VDI is not where the savings are, but the operational costs are where all the savings come from.  I am sure there are some fancy marketing charts or Gartner charts that attempt to prove this, but let’s take them at their word.  The challenge here is that most law firm’s would not be replacing their local desktops (with Thin Terminals) should they move forward with a VDI solution.

Honorable Mention

Capital costs for shared storage and virtualization infrastructure, Thin environments with XenApp/Terminal Servers can be built for a fraction of the cost of VDI.

Conclusions

I am going to end this post by saying the success to any VDI environment (or any technology project for that matter) is knowing your use case and business requirements.  This has been drilled into me by my colleague Matt Liebowitz who probably starts every VDI conversation with the phrase “know your use case”.  Too many times technologists lose sight of the business requirements and use cases for a particular technology before attempting to fit a square peg in a round hole.  VDI offers obvious benefits in some areas over a traditional desktop, but it isn’t a one size fits all model.   The use case I focus on in this post (VDI as a desktop replacements) relay my thoughts on why most firms see VDI as an non starter because the challenges associated with implementing VDI outweighs the firm’s business requirements for doing it.  Is VDI a non starter for all law firms?  Absolutely not. There are some scenarios in which a firm’s business requirements and existing desktop circumstances make VDI an attractive and potentially more cost effective approach than a traditional desktop.

The requirements to participate in the Microsoft Partner Program have recently evolved to help differentiate technical and business capabilities among participants; Kraft Kennedy has risen to the challenge by quickly exceeding the goals set forth by the program.

Each competency requires specific individuals with deep technical skills, Microsoft verified customer references, and challenging certification exams to be completed.  This commitment demonstrates our breadth, deep specialization, and proven expertise across a range of Microsoft technologies.

About Kraft Kennedy

Kraft Kennedy provides business and technology-related consulting services to the legal community. By combining outstanding technical skills with an intimate knowledge of our clients’ business and information needs we tailor solutions that enhance attorney productivity, effectiveness, and client value.

We focus on the business needs of the client and ensure that technology is used to enhance, not inhibit their business. KK’s talented staff of strategic consultants, project managers, and network consultants have years of experience with hundreds of projects for firms from small to large. Our services portfolio includes advanced infrastructure projects, business continuity and data center consolidation, desktop deployment, network design and implementation, storage design and replication, and messaging systems migration among others. Our Microsoft specialties include: Desktop, Server Platform, Unified Communications, Portals and Collaboration, Search, Systems Management, Virtualization, and Small Business Specialist Community.

The Citrix Certified Enterprise Engineer for Virtualization certification is designed to certify breadth of knowledge across Citrix virtualization solutions. The CCEE offers experienced IT professionals the opportunity to develop and validate their ability to combine the coordination of operational planning efforts with tactical design expertise and integration know-how.

Niraj and Jeff have a well established reputation for helping law firms adopt thin-client technologies and best practices around enterprise desktop and application management. It is exciting to watch server-based computing strategies mature as new virtual desktop solutions, such as VDI and application virtualization, have become more mainstream. Not only are these new tools offering firms technical improvements, but also significant operational efficiencies and flexibility over traditional desktop and server-based approaches.

On behalf of the team, Congratulations to Niraj and Jeff on your accomplishments!

First and foremost, XenClient has a very strict HCL.  It is specific to the most modern Dell, HP and Lenovo’s business series laptops with Intel Graphics.  This isn’t a surprise and is actually expected to some degree.  I know that Citrix is working closely with OEMs and Intel for compatibility, so this will expand as the product matures and if it finds a niche.  I was working with a test Dell Latitude E4300 as my XenClient laptop and was able to complete the install within 10 minutes.  It was very straightforward and went without a hitch.  The hypervisor takes about a minute to boot up and then you are presented with the main GUI (screenshot below).  The UI is pretty straightforward, and includes functional information like remaining battery life, gives the ability to connect to different wireless networks, and configuring some basic hypervisor’s properties.

Using the wizard driven menu, I created a Windows 7 desktop and that is what was presented at the center of the screen.

After Windows 7 was installed in a VM, I installed the XenTools, but still had an issue with audio playback.  This was resolved easily by installing the Realtek AC97 audio driver.  My test laptop only had 2 GB of RAM on it and although the hypervisor was utilizing 600+ MB of RAM, the 1 GB Windows 7 desktop ran pretty well.  I confirmed the hardware audio buttons worked and functions passed through to the VM and USB devices such as iPhones and aircards were also passed through without error.  Also, because the XenClient supports paravirtualization, it is able to directly pass through the Intel video card to the running VM.  This means Aero support, HD video playback among other functions that require video acceleration.  Check out the screenshot from the running VM.  The Intel GMA graphics driver is passed through into the virtual machine.

Finally, there is the “Synchronizer” piece that is the server backend that the XenClient can connect to.  It allows deploying, updating and managing virtual machine images centrally against connected XenClient hosts.  The Synchronizer Virtual Appliance only supports XenServer at the moment, so it has been a non-starter for me and expect it will be in most other environments as well.  It is rumored that a virtual appliance that supports ESX is slated for release later this year, so that should help make it easier to test.  I’ll do more testing when this is made available.

Anyway, the “1.0” release of XenClient should really be called an extended beta.  The product is very slick, but is far away from any production use.  I would go so far as to say that we are at least 12 months away from XenClient seeing a production environment.  Citrix has a interesting product, but has the challenge of figuring out specifically how XenClient fits into existing XenDesktop and Provisioning Server environments.  Will the Synchronizer be integrated into Provisioning Server in the future?  Will a Provisioning Server vDisk have the ability to be offline with XenClient?  XenClient introduces a lot of questions.

In my recent post on the effects of ASLR in vSphere the comments turned into a discussion about TPS on modern processors. And there are countless posts about this issue on the VMTN forums where folks are looking for a fix. In reality nothing is broken and there is no need to fix the issue.

VMware has published a KB article that gives more information on TPS with Nehalem processors and why it appears TPS isn’t working (this affects modern AMD processors also). The short version is that TPS uses small pages (4K), and Nehalem processors utilize large pages (2MB). The ESX/ESXi host keeps track of what pages could be shared, and once memory is over-comitted it breaks the large pages into small pages and begins sharing memory.

Many people think this is a bug in ESX that needs to be fixed. This likely started because when vSphere 4 was released there was a bug around memory usage on ESX hosts with Nehalem processors. In reality the bug was that vCenter was triggering high memory usage alarms for virtual machines running in this configuration. Nothing was actually wrong but because the host was using all of the assigned memory for the VM, vCenter was incorrectly triggering the alarm. That behavior has since been fixed with a patch and is no longer an issue.

So what does this actually look like? When a VM is powered up on an ESX host with Nehalem processors, the amount of host memory in use will not drop down as the VM uses less memory or becomes idle. Those of us that have been using ESX for a long time likely found this scenario disturbing.

From vSphere Client (red highlighted section shows guest taking all of the 2GB assigned memory, yet memory usage in the guest is very low):

From esxtop (red highlighted section shows almost no memory being shared with page sharing):

The above screenshots show a host that is under-committed on memory and so no page sharing is occurring.  If the host gets over-commited page sharing kicks in automatically by breaking up large pages into small pages.  You can force the use of small pages on all guests all the time by changing the value of the advanced option Mem.AllocGuestLargePage to 0.  I don’t really see any reason to do this – remember that TPS isn’t broken and what you see in the above screenshots is normal and expected.

Once host memory is over-committed (or if you use the advanced option), memory sharing kicks in and things look like they normally do when page sharing is taking place.

From the vSphere Client (red highlighted section shows guest taking very little of the assigned 2GB memory as page sharing has kicked in):

From esxtop (red highlighted sections show a large amount of shared memory and the host is over-commited on memory by 48%):

A quick note on the esxtop screenshot above – it was taken from a VDI environment where all workloads are identical so that explains the high amount of shared pages.  It was also overcommitted more than normal as it was taken during host maintenance.

I hope this clears up some of the confusion around TPS on modern Intel/AMD processors.  In short, don’t get hung up on the fact that TPS isn’t kicking in like it did with older processors.  Nothing is broken, TPS is working as expected, and it will kick in when you actually need it.

Citrix NetScaler VPX

• Released last year in Q4.  Initially, this virtual appliance was only released for XenServer, but now has full support for ESX.  Expected to support Hyper-V in the late summer.
• Licensed by throughput and available in 10 Mbit, 200 Mbit and 1000 Mbit flavors.
• Free “Express” edition with 1 Mbit throughput limitation available for lab and PoC environments.
• Standard NetScaler “Advanced”, “Enterprise” and “Platinum” flavors available.

Citrix Access Gateway VPX

• Released earlier this month.  Currently only supports XenServer. Support for ESX and Hyper-V expected in the next 6-12 months.
• Provides same feature as Model 2010 Access Gateway physical appliance.
• Free “Express” edition exists that provides access to 5 concurrent users that are valid for 12 months.

Citrix Branch Repeater VPX

• Released in Q1 of 2010.  Currently only supports XenServer. Support for ESX and Hyper-V expected in the next 6-12 months.
• VPX Appliance does not support the following features available in the physical appliance:
  • Group Mode
  • Ethernet bypass card
• Still requires Citrix Repeater appliance, which is not available in VPX format.

XenClient 1.0 will be released later this week for download on MyCitrix and is being demoed and talked about at the conference.  After playing with it at one of the demo stations and talking with some Citrix Engineers, here are some details:

• Unlike VMware’s View, XenClient is a Type 1 hypervisor.  This means it lives above the client side OS (Windows).  Once installed, the user has the option to boot into whatever VMs are available on the laptop.
• At the moment, XenClient will only support a small subset of hardware types.  This includes Dell’s Latitude E series, Dell Optiplex 780, and  HP EliteBook laptops.  Full HCL to be published later in the week.
• Citrix Synchronizer is the server appliance that chats with the XenClient (over HTTP/SSL) to continuously sync and update the local running VM back to the Data Center.
• Synchronizer will be available as a virtual appliance running on XenServer.  According to Citrix, there is no planned version for ESX.  – I am sure this will change though.
• XenClient can be installed as a standalone or in conjunction with “Synchronizer”.
• XenClient supports paravirtualization to allow VMs direct access to hardware (using it’s native driver).  For example, a VM under XenClient can tap directly into a GPU for accelerated video playback and graphic intensive applications.  – This demoed very well with the engineer playing back an HD video file without skipping within a VM.
• At the moment, only a small subset of USB devices are supported through XenClient.

I am at Synergy all week, so I expect to learn more details about XenClient over the course of the week.  If you guys have questions, feel free to post them in the comments and I’ll try to get those questions answered while at the conference.

I’m working on a vSphere implementation using Dell EqualLogic SANs and wanted to configure Round Robin on all of my datastores. Dell has a great whitepaper on how to set this up, but unfortunately the document fails to mention one key thing: this doesn’t change the default path selection plugin (PSP) from Fixed to Round Robin.   That means that you’ll have to set the multipathing policy to Round Robin on all of your existing datastores and will have to remember to do that on all future datastores. When you’ve got multiple ESX hosts with lots of  datastores this can quickly become a pain.

Luckily there is a way to force the default multipathing policy to Round Robin. The following commands can be used to change the default PSP to Round Robin as well as configure round robin specifically for the EqualLogic provider.  These commands can be entered at the Service Console or via the vSphere CLI 4.0:

esxcli nmp satp setdefaultpsp –satp VMW_SATP_DEFAULT_AA –psp VMW_PSP_RR
esxcli nmp satp setdefaultpsp –satp VMW_SATP_EQL –psp VMW_PSP_RR
esxcli corestorage claimrule load
esxcli corestorage claimrule run

Note that “satp” and “psp” are preceded by two dashes and not a single dash as it appears in this blog post.

Once you enter those commands (no rebooting required) any volume you add, either new or existing, will use Round Robin MPIO by default.

Address Space Layout Randomization (ASLR) is a security feature that randomizes the position of data in memory, making it more difficult for attackers to predict where data can be found while in memory. This feature has been enabled in Windows since Windows Vista, and other operating system such as Linux and MacOS implement this in some form as well.

Since ASLR randomizes information in memory it makes sense that it would be more difficult for TPS to find identical memory pages and thus memory sharing would be reduced. But just how much of a difference does it make? I decided to try and find out. Here are the specs from my test environment:

Server: HP DL385 G1 (AMD Opteron 275)
ESX: 4.0.0 build 244038
Guest OS: Windows Server 2008 R2
Guest RAM: 2.5GB

All guests were cloned from the same template and have the same software installed. On guests TESTSRV1 and TESTSRV3, I left the default settings. On TESTSRV2 and TESTSRV4, I disabled ASLR using the following regkey:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management]
“MoveImages”=dword:00000000

In all of my testing, including leaving VMs idle and also running memory tests, I found no measurable difference in the amount of memory shared with TPS. I also looked at esxtop to see how much memory was actually being shared and I saw virtually no difference whatsoever between VMs that had ASLR enabled and those that had it disabled.

Host Memory Usage:

esxtop statistics:

The SHRD and SHRDSVD columns represent how much memory is being shared with TPS and the total memory savings. Clearly there is quite a bit of memory sharing going with or without ASLR enabled.

Why would this be the case since it makes sense that TPS would be hurt by ASLR? ASLR requires applications to “opt-in” to have their memory randomized, and I suspect that much of Windows 2008 R2 is not opted in. Perhaps applications will come out in the future that are written to take advantage of ASLR, but at the moment that doesn’t appear to be the case.

Of course this is by no means a definitive test as it wasn’t run with production systems and real users running real applications. That said, I think it shows that ASLR does not dramatically reduce the amount of memory shared with TPS. I did also look at production systems left at the default settings (ASLR enabled) and saw similar memory sharing gains. I’m curious if others have seen similar results in their environments, so drop me a line if you’ve done any similar testing.

More info:

What is ASLR (Wikipedia)
Interpreting esxtop statistics

When virtualization became popular it presented a challenge to those looking to continue to use their tape drives in fully virtualized environments. If you were using VMware you could use SCSI pass-through to present a tape drive or changer directly to a virtual machine but that prevented you from using any advanced features like VMotion. It also tied your tape drive and VM to a single host containing a SCSI card, making things complicated if that host were to experience a hardware failure.

While this is still possible in vSphere 4 (and the previous version), this configuration is not ideal.  Instead, consider converting that SCSI tape changer into an iSCSI target that can be used on any virtual machine attached to any host by using an iSCSI-to-SCSI bridge.  These bridges let you attach your tape changer directly to the device and then present the tape changer to virtual machines as an iSCSI target.  There are several different vendors providing this technology, including Atto Technology, Paralan, and others.

Once the tape drive is attached to the iSCSI bridge and configured as a target, you simply use the Microsoft iSCSI initiator inside a virtual machine to connect to the device.  The tape device will appear to the virtual machine as if it were any other iSCSI target (like a SAN LUN).

After connecting to the target in the iSCSI Initator, the tape device will become visible in Device Manager on the VM and within tape backup software.

After the tape device has been successfully discovered, the virtual machine can then be managed with features like VMotion, HA, and DRS because the VM won’t be tied to an individual host.  This configuration also opens up other design possibilities, such as multiple backup servers running different backup products.  Using the iSCSI bridge provides a lot more flexibility than directly attaching the tape device to an ESX host.

Sounds great, right?  As always, there are things to consider before moving forward with this type of solution:

1) Does your backup software vendor and tape changer vendor support this setup?

2) Will this setup meet the performance requirements of your environment?  In practice I’ve seen these devices push 2 GB/min or more, similar to the performance of direct attached tape devices.

Using an iSCSI-to-SCSI bridge opens up a lot of possibilities for keeping a tape device in your fully virtualized environment.  It also simplifies your setup and allows you to take advantage of enterprise features of your virtualization product.  Finally, for around $1,200 for the iSCSI bridge this is also an affordable solution to a common problem.