Welcome to Kraft Kennedy

For years organizations have relied on tape drives and changers for backup and recovery of their critical data. Despite many predictions to the contrary, tape is still alive as we begin 2010.

When virtualization became popular it presented a challenge to those looking to continue to use their tape drives in fully virtualized environments. If you were using VMware you could use SCSI pass-through to present a tape drive or changer directly to a virtual machine but that prevented you from using any advanced features like VMotion. It also tied your tape drive and VM to a single host containing a SCSI card, making things complicated if that host were to experience a hardware failure.

Continue reading…

My last post Citrix Provisioning Services Part 1 – What Is It? served an introduction to what exactly Citrix Provisioning Services is capable of. Below I hope to open people’s eyes to using PVS for something other than VDI, as it is often thought of as a part of the XenDesktop suite. However PVS is actually independent of XD or VDI, and can be utilized in combination with XenApp to bring single-image benefits to the Terminal Services world.

Provisioning Services allows for server consistency, easier maintenance, dynamic servers, and aids in disaster recovery.

• Consistency – As a best practice every XenApp server delivering the same applications should be 100% identical to the rest of the farm. However, obtaining this is easier said than done.  By streaming the same image to every server, each server is inherently and 100% the same as the rest.
• Maintenance – Updating and patching large farms can be a very time consuming task, and anything done to one server must be repeated for the entire farm to maintain consistency. With PVS, patches and software installations are applied once to the master image and on next reboot, each XenApp server boots the new updated image. In addition to software patching and installation, Terminal Servers need to be completely refreshed periodically to keep them clean and performing optimally; they are used by dozens of different users, reducing performance and resulting in inconsistent servers. A typical server refresh requires the server to be re-imaged and the software redeployed, a time consuming process that can be prone to error, leaving a server in an unusable or inconsistent state. Operating system streaming with PVS results in a completely fresh and optimized server on every reboot.
• Dynamic – PVS allows for a dynamic XenApp farm instead of a static one. As load rises and additional servers are needed, they can be quickly brought online in seconds instead of hours. Conversely, as load drops, un-needed servers can be powered off or repurposed as needed. A server becomes a vessel for different workloads and can be a XenApp server one day and an IIS server another if need be. Since Provisioning Services is capable of streaming to both physical and virtual servers, administrators have the ability to utilize different types of resources all from the same master image(s).
• Disaster Recovery – Creating a disaster recovery plan for the XenApp environment often requires complex processes, scripts and configurations. Assuming a PVS server has been built in the DR site, and that the master image has been replicated as well, quickly bringing an entire farm of XenApp servers online becomes a simple task.

Creating a XenApp environment that is more dynamic and easier to maintain is a goal for many XenApp administrators. The addition of Provisioning Services to a XenApp implementation can go a long way to achieving those goals. By leveraging the single-image management capabilities of PVS, administrators can dramatically reduce the costs involved with deploying and maintaining their XenApp farms. While at the same time, guaranteeing consistency between and ensuring peak performance of each server in the farm. All while being capable of quickly adapting to changes in load and disaster scenarios.

One of the great features of desktop virtualization (VDI) being touted by the industry is the ability to manage and update all of your desktops from a single central master image.

Citrix’s solution to the single image process is accomplished by a product called Provisioning Services (PVS). This software is the result of their purchase of a company called Ardence back in 2007. Provisioning Services is an often misunderstood piece of software, and its great benefits and potential are not necessarily apparent to everyone.

PVS works by streaming a master (read-only) image from the server to a target server or workstation. Any subsequent writes are then sent back to the PVS server and written to a cache file. The reads and writes are sent back and forth between the PVS server and target in a constant stream over the network. The easiest way to grasp this is to imagine that the cable connecting the hard disk inside of the server to the motherboard (and thus the CPU and RAM) is replaced by a network cable running back to the PVS server. The operating system sees the PVS disk as though it were a normal hard disk, and everything is done entirely transparent to the OS. The magic happens when the server is powered up; instead of booting from a local disk it is instead set to boot to the network card (PXE, BOOTP) which talks to a service on the PVS server, which streams the assigned operating system image to the target. The target device starts up immediately, as though it was booting from a local disk.

The beauty here is that this single read-only image can be simultaneously streamed to multiple diskless targets, both physical and virtual. This central image can now be maintained in one place. This makes tasks such as installing updates or new software quick and easy. After installing an update into the master image, all machines running that image will boot up into the updated image on next restart. To put that in perspective, think of the time and effort required to push out something such as a service pack to Windows or Microsoft Office to your entire firm. Now imagine simply installing that update once and having every machine in your environment receive that update on next reboot, without any additional effort.

Look for a follow-up post discussing the benefits that Provisioning Services can bring to a XenApp implementation.

With the release of VMware vSphere 4, VMware has released a very powerful management tool called Fault Tolerance (FT).  At a basic level, FT allows you to keep two virtual machines (a Primary VM and a Secondary VM) running in lockstep on two different physical ESX hosts.  If one of the ESX hosts were to experience a hardware failure, the VM protected with FT would remain running on the second host without any downtime.  This can greatly reduce downtime due to hardware failures and provide increased service levels for important applications.

FT is often compared to Microsoft Windows Failover Clusters, formerly Microsoft Cluster Server (MSCS), and in fact many have talked about how FT can replace Microsoft clustering altogether.  Rather than jump to conclusions like this, it is important to understand the use cases for both technologies.  In addition, there are several limitations to FT that need to be considered. Here are some important points to remember about FT:

Continue reading…

In a previous post, I discussed how to expand virtual machine boot volumes with no downtime using Dell’s ExtPart utility.  Using this method is useful if you are using Windows Server 2000/2003, but is no longer required when using Windows 2008.  With Windows 2008 becoming the preferred server operating system going forward, using the method described below will become more and more common.

Microsoft has made it possible to expand boot volumes easily and on the fly without downtime without the need for any additional tools.  In this example, we’ll expand the C: drive of a Windows 2008 virtual machine from 25GB to 30GB.

1. Expand the size of the virtual disk.

2. Launch Disk Management by right clicking on My Computer, selecting Manage, and then selecting Storage\Disk Management.  If you do not see the unallocated space, right click on Disk Management and select “Rescan Disks.”

Continue reading…

Installing the VMware Tools package inside a VMware virtual machine improves overall performance and allows the use of advanced features and faster virtual hardware drivers.  The installation package also installs a tray icon that controls guest access to virtual hardware, time synchronization, etc.  Since most virtual machines are servers and end users don’t typically access the console of a server, worries about the security implications of leaving that tray application running have been fairly minimal.  However, as firms move towards solutions like virtualized XenApp servers or virtual desktops, this becomes more of a concern.

Removing administrator access to end users is unfortunately not enough.  For example, a user can open the VMware Tools tray icon and select the Devices tab, and from there can uncheck “NIC1″ and click Apply.  What happens?  You guessed it – the virtual NIC is disconnected and the user loses connection.  That’s bad in a virtual desktop environment since it will orphan the desktop and likely require a connection broker like XenDesktop to create another desktop but it is even worse on a XenApp server where the user potentially just disconnected dozens of other users as well.

This, and several other things found in the VMware Tools, can be dangerous to leave available to an end user even if they have no rights to the server itself.  To get around this, there are two approaches that make sense:

1) Remove access to the VMware Tools for end users.

2) Modify the VMX configuration file to prevent these actions.

I prefer the second method since it allows for more granular control over security, though if you’re interested in option one then you can read VMware’s KB article on the subject.  In order to prevent this at the VMX (virtual machine configuration file) level, simply add the following lines to the virtual machine(s) that you wish to protect (after powering it down):

isolation.device.connectable.disable = “true”
isolation.device.edit.disable = “true”

To see how to add one of these values to the VMX file via PowerShell and PowerCLI, it would look something like this:

$vm = Get-View (Get-VM NameofVM).ID
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmConfigSpec.extraconfig += New-Object VMware.Vim.optionvalue
$vmConfigSpec.extraconfig[0].Key=”isolation.device.connectable.disable”
$vmConfigSpec.extraconfig[0].Value=”true”
$vm.ReconfigVM($vmConfigSpec)

There are many other security parameters that can be set in the VMX file that are covered in VMware’s Security Hardening document (PDF).  The document covers this and many other common security best practices for virtual machines.  As always, test any change you make (especially the script above) before putting anything into production.

Microsoft has done a nice job of late regarding supporting their operating systems and applications when they are run in a virtual machine. First they created the Server Virtualization Validation Program to validate their software running on hypervisors from various vendors, including their own Hyper-V.  They’ve taken the SVVP one step further by adding a new tool called the Support Policy Wizard that makes it fast and simple to verify Microsoft support.  It can be found at the following link:

http://www.windowsservercatalog.com/svvp.aspx?svvppage=svvpwizard.htm

This tool makes it easy to select a specific configuration, including operating system, hypervisor, and Microsoft application, and then verify it against the SVVP to ensure Microsoft supportability.  It also lists any specific application features that are or are not supported.  Validating your configuration against Microsoft’s Support Policy Wizard is an important step, especially when virtualizing mission critical applications like Exchange or SQL.

Try the wizard and see the results for yourself.  I tried it myself and the results are below.  I selected Exchange Server 2007 SP1 running on vSphere 4 with Windows Server 2008 x64 and received the following supportability statement back:

Summary Support Statement*
This configuration is Supported.

* Customers with Premier-level support agreements should contact their account manager for more information
* Additional information is available in the “Support policy for Microsoft software running in non-Microsoft hardware virtualization software” which can be viewed here

Support Statement Details
Product: Exchange Server 2007 Service Pack 1 on VMware vSphere with Windows Server 2008 (x64) Guest OS

Search the Knowledge Base for information related to this configuration

For Exchange Service 2007 Pack 1 and later, see here for specific configuration information. Note: The Exchange Server guest virtual machine must be deployed on the Windows Server 2008 operating system.

Specific third-party virtualization information is available at here

To get support on the virtualization solution, the customer also needs to have a support agreement with the third party vendor.

Supported features: Anti-Virus, Back-up Software, Virtual Machine Management Software, Cluster Continuous Replication (CCR), Virtual Processors

Unsupported features: Unified Messaging, Dynamically Expanding Virtual Disks, Virtual disks that use differencing or delta mechanisms, Hyper-V Quick Migration combined with Exchange Clustering, Virtual Machine Snap Shots

One of the great things about using virtual machines is how easy it is to quickly modify their hardware configuration.  One common change that administrators often make is to expand the size of the virtual machine’s hard disk.

Windows data volumes are easy to expand by simply growing the disk file and then using Microsoft’s Diskpart utility to make the extra space visible in the guest.  This can be done on the fly with no downtime to the virtual machine.

For boot volumes it is more complicated and is typically handled in one of two ways. First, the admin expands the boot drive of the virtual machine to the desired size. Then the VM is shut down, and one of the following two methods is used to expand the volume:

1. The expanded hard disk is attached to a second virtual machine, and then Diskpart is used to make the new space usable in the guest.  That second VM then has to be shut down, the disk removed from the configuration, and both servers booted back up.
2. The VM is booted with a GParted ISO which allows the volume to be expanded.  Windows will force a chkdsk run at next bootup but will then see the expanded size.

While these are acceptable solutions, both require downtime of one or more virtual machines.  To get around the downtime requirement, you can use a utility called ExtPart from Dell. It is used to expand partitions on Dell storage arrays but works just as well in a virtual machine. Note that it can only be used on Basic disks in Windows and does not work with Dynamic disks.

The following example shows expanding a 17GB C:\ drive to 20GB on a Windows 2003 VM with no downtime required.  Note that this process is not required on Windows Server 2008 since the ability to grow volumes is now a feature of Disk Management.  Simply right-click the volume you wish to expand and select Extend Volume.

1) Edit the virtual machine settings and increase the boot volume to the desired size.

2) Launch Disk Management and select “Rescan disks” to detect the new space.

3) Run ExtPart with the following syntax: extpart.exe <volume> <AdditionalSizeInMegabytes>

After completing steps 1-3, the new space should be recognized in Windows automatically.

Discovering that your ESX hosts have been unknowingly creating open snapshots can be an alarming, not to mention dangerous, event.

Third party storage and backup vendors frequently call the vCenter API to issue a snapshot creation of running virtual machines before they grab a copy of the virtual machine’s hard disk – VMDK files – which are located on the VMware VMFS datastore. This process is typically followed by an immediate deletion of the snapshot file which will merge the changes back into the initial VMDK file. Problems can occur when this process does not complete successfully leaving you with an “open” snapshot. This problem is compounded when it occurs multiple times against the same guest.

Each open snapshot can significantly degrade virtual machine performance and also contribute to poor storage utilization. Unless you monitor each of your virtual machines on a daily basis, you could quickly be left with an uncomfortable situation on your hands.

Continue reading…