While the details of this specific seizure are clear, this event should serve as a reminder of one of the risks associated with cloud/SaaS/Hosted providers. If the government targets a service that your firm uses, all of the data you store could disappear in an instant.  Possible services include file sharing services, firm website hosting services, online backup systems and even nfrastructure as a service firms.   When evaluating the cloud these concerns should be taken into consideration and understood.  Your firm might not be doing anything wrong, but could suffer greatly if someone else is and the government decides to intervene.

The first step is to create a contact that will serve as the recipient for the SMS message.  You will need to consult the settings for your particular carrier to find the format for this address.  For AT&T users, the format will be [10-digit phone number]@txt.att.net.  Once the contact is created, create an email rule similar to the one below:

This rule will tell Exchange to forward any message marked as “High Importance” to the contact that was created for the SMS address. While the above view is from a Windows Outlook client, the same can be accomplished through OWA, or in Entourage or Outlook 2011 for Mac.

Within a few minutes (depending on your reception and your carrier), you will receive a text version of the email message that was marked with high importance.  Because of the inherent 160 character limitation of SMS messaging, it is unlikely that you will be able to read the entire message, however the SMS message will alert you to the fact that you have a priority email and you can then read it in the native iOS email client. Keep in mind, the source of the text message may vary based upon your carrier. Here is how the text message may appear on an AT&T connected device. Note the highlighted portion of the text.

One thing to be careful of… Normal text messaging rates will apply, so if you receive a large number of priority messages, and do not have an unlimited messaging plan, this could get expensive very quickly.  You can lower the number of messages that get forwarded by tweaking the rule to filter for certain senders, but its always a good idea to be aware of your SMS plan.

Citrix provides a nice PowerShell command to query the RAM cache used via the MCLI PowerShell snap-in.  This snap-in (McliPSSnapIn.dll) comes standard with Provisioning Server Console install and is located in C:\Program Files\Citrix\Provisioning Services Console.  Before it can be added to PowerShell, the dll must be registered on any system wish to run the query from.

To register the dll on a 64-bit machine, run the following PowerShell command:

• C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\InstallUtil.exe ‘C:\Program Files\Citrix\Provisioning Services Console\McliPSSnapIn.dll’

To add the snap-in to a PowerShell command prompt, run the following command:

• Add-PSSnapin -Name McliPSSnapIn

Run the following command against a XenApp server to see how much RAM cache is used:

• mcli-get deviceinfo -p devicename={servername} -f status

The second number returned in the status section is the percentage of the RAM Cache used.

While there are a number of hotfixes and other items included, there is also some key new functionality being introduced as well:

1. Cross-Site Silent Redirection for OWA – This is a tremendous improvement over current functionality where, if a firm leverages redirection between Client Access Servers in different sites (the preferred approach for optimal performance and implementation flexibility), users are prompted with a link and second authentication prompt if they login to a CAS server in a different site than where their mailbox is currently hosted.  With this new functionality, Exchange can be configured silently to redirect users to the correct CAS server in this situation (without reauthentication or prompting).
2. Address Book Policies – Address Book Policies provide the long-awaited native functionality support for what is typically referred to as GAL segmentation.  Previously, if firms wanted selectively to exclude some users or contacts from the GAL for specific subsets of users, the firm would need to create and manage security ACLs directly within ADSI.  This was unsupported in Exchange 2003 and 2010 and only narrowly supported for Exchange 2007 (with Dave Goldman’s specific whitepaper).  Address Book Policies will provide an object and policy based method for providing this functionality.
3. OWA Mini – This will be a lightweight, text-only version of OWA targeted for use on mobile devices or in low bandwidth/resolution scenarios.
4. Hybrid Configuration Wizard – This wizard will significantly reduce the number of steps required to streamline the process for establishing rich coexistence between an on-premises Exchange 2010 environment and Office 365 (formerly BPOS).

Due to the nature of the new features included, there is an Active Directory schema update required for SP2.

Great news!

As we approached our anticipated IT pre-pilot for the new Exchange 2010 environment, we started to notice significant issues in DAG communications across the WAN.  Specifically, we saw the following issues fairly consistently although, at some times, everything worked just fine:

• From the primary data center, viewing the mailbox database and associated copy status from the Exchange Management Console listed mount states for some databases as “Unknown” and copy status for all remote database copies as “ServiceDown.”  Running Get-MailboxDatabaseCopyStatus against the DAG member(s) in the remote data center reflected the same results.  Databases in an “Unknown” mount state corresponded to cases where the database was activated in one data center and status was being queried across the WAN from the other data center.
• Running “Get-DatabaseAvailabilityGroup -Status” would take an extremely long time to complete.
• Occasionally, databases would be listed in a dismounted state and, upon attempting to mount, an error message stating “Automount consensus not reached” would be returned and the mount would fail.
• Event logs on DAG members in both data centers would report sporadic occurrences of FailoverClustering events reporting that nodes in the repsective remote data center had been removed from cluster membership.
• Test-ReplicationHealth against DAG members across the WAN to the remote data center reported failures for ActiveManager (“Active Manager is in an unknown state”) and TasksRpcListener (“An error occurred while communicating with the Microsoft Exchange Replication service to test the health of the Tasks RPC Listener”).

The issue was clearly related to RPC requests traversing the WAN and having issues somewhere along the path from source to destination.  As a next step, I ran the “Validate a Configuration Wizard” for the DAG’s underlying Windows Failover Cluster and, sure enough, RPC errors were reported for queries that needed to cross the WAN to talk to cluster nodes not in the same data center as the node on which the wizard was run.  At this point, it was time to install Wireshark and run packet captures on either side of the WAN while executing Exchange actions or the cluster validation wizard to determine what was happening to the traffic.

Upon review of the packet captures, it was revealed that packets were being sent between DAG/cluster members that were larger than a standard 1500 byte packet and those packets were being fragmented in transit from source to destination.  Disabling various large TCP offload functionality of the NIC driver in use within the DAG/cluster members (vmxnet3 Ehternet Adapter) helped to bring the packet size down to 1500 bytes but the problems still occurred.  Running ping tests between the two data centers (ping -f -l <packet_size> hostname) revealed that the largest packet succeeding across the WAN was 1468 bytes.  Once the MTU of the NIC was reduced to match this value (via NETSH), everything began working perfectly.  The “Validate a Configuration Wizard” for the cluster completed without any unexpected warnings and all Exchange-related functionality was restored.

While disabling various functionality on the NIC and reducing the NIC’s MTU worked to solve the problem, it was certainly not ideal nor a long term solution for this environment.  Ultimately, determining where the issue lied in the WAN environment was key to identify how to resolve this issue without requiring non-standard configurations on various servers in the environment.  In working with the client’s networking team, it was understood that their particular WAN connectivity provided a Layer 2 Ethernet hand-off to each data center such that no router was in place on either side.  This explained why larger MTU packets were traversing the WAN and being fragmented in the process.  Coordination between the WAN provider and the remote data center’s networking team was required to determine where in the network path a device was unable to handle even a standard 1500 byte packet properly.

Ultimately, there were a few options for remediation at this client in the form of either obtaining jumbo frames support across their Layer 2 WAN or placing routers on either side of the WAN.  Both of these options would remove the requirement for non-standard configurations on the actual servers while still resolving the issue of communications between the data centers.

First, let’s take a quick look at the MAKtivation process.  Before any target devices are activated, the following option is available in the Provisioning Server console:

Selecting that option brings up the following dialogue:

Assuming you’ve installed Microsoft VAMT 2.0 on all Provisioning Servers, simply enter your MAK key, click OK, and you’re off to the (somewhat slow but effective) races.  When all target devices are activated you’ll receive the following confirmation, and your target devices should be free of any Windows activation warnings.

You’ll also notice that the option to “Manage MAK Activations” no longer appears in the Provisioning Server console:

This process worked well until I upgraded the VDA in my vDisk to support XenDesktop 5.5.  After rebooting, Windows reported that hardware had changed and that I needed to reactivate.  Unfortunately no one told Provisioning Server because the MAKtivation option was still gone, temporarily leaving me in unactivated and unable to reactivate purgatory.

Per Citrix’s guidance, an automated vDisk update process must be followed in order to preserve MAKtivation.  Yeah… so we’re gonna have to go ahead and break that on purpose by manually disassociating the vDisk from the affected target devices:

Ensure your target devices are powered off and confirm that the vDisk associations are removed in the console:

Once that’s done, manually reassociate the vDisk, boot your target devices, and *POOF* the MAKtivation option is back:

Hurricanes are unique in terms of natural disasters in that we usually have some warning before they strike and can prepare accordingly.  It’s important to plan for a hurricane as an extended event and not just plan for the storm itself, as it can take weeks to undo the damage to local infrastructure.  With that in mind, I’ve put together the following recommendations for steps to take to protect your firm’s data and infrastructure:

• Communicate with the firm’s staff. Setting the right expectation of what measures are being taken to protect the firm’s IT infrastructure goes a long way.
• Ensure you have a good, recent backup available. Make sure the backup in stored in a safe, waterproof location, and ideally one that is accessible under emergency conditions.
• Make sure firm staff know all their remote access options. This will allow users to continue to work if they can’t physically make it to the office but the systems remain online.  Provide them a printed copy of the remote access information to have on hand.  Also remind staff to take home laptops and mobile broadband cards when they leave for the weekend. If they will be connecting to an e-mail continuity service, make sure they know the proper login credentials
• Review your disaster recovery options. If you have a reliable DR option, it may be a good idea to put it in place before the actual event.  Doing so after the storm in the event of an outage may prove much more difficult.
• Shut down as many systems as you can. This will obviously vary based on your firm’s business requirements and your faith in your automatic shutdown methodology.  If you’ve tested your UPS solution recently and know it will bring systems down gracefully in the event of a power outage, you probably don’t need to go this route; if you have any doubts, consider a preemptive power-off to prevent a hard shutdown of your entire network.  Your site may not lose power at all, but if it does and you don’t have a backup generator the outage will likely far outlast your UPS’s capacity.
• Ensure you have the ability to remote back in when power is restored. Review your VPN and out-of-band access if you haven’t used them in a while to make sure everything is still configured properly.
• Move exposed hardware out of harms way. Depending on where your IT infrastructure exists in your building, it may be in danger of being exposed to the elements.  Get hardware as far away from windows as possible, cover it if you think there’s a chance of it getting wet from above, and elevate it if it’s located on the first floor or basement.
• Power off any unprotected devices. This is especially important with your workstations, which often are plugged directly into wall sockets.  If you want to go the extra step, physically unplug devices from their power source.  Power surges during a hurricane are notorious for damaging electronics.
• Keep your cell phone plugged in whenever possible. Instruct your IT staff to do the same.  You’ll be grateful that you did if the power actually goes out and you need to get in touch with someone.
• Keep a printed emergency contact list with you at all times. Don’t rely on the contact list on your electronic devices.

AND MOST IMPORTANTLY:

• Obey any local emergency notifications and evacuation orders. I can’t stress the importance of this point enough.  If you’ve been told to evacuate, there’s a very good reason.  By sticking around, you’re likely putting yourself in harm’s way.  Stay away until you’ve been told it’s safe to return.

These storms are highly unpredictable and highly dangerous.  Hurricane Irene could change course and miss the Northeast entirely, or it could be the storm we talk about for years.  Either way, the best thing to do is to take measures as far in advance of the storm as possible and keep yourself safe when the storm actually hits.

-Danny (in addition to being a Senior Technical Consultant at Kraft Kennedy, Danny is a firefighter for the Northwest Volunteer Fire Department in Houston, TX. Ed.)

PS – Check here for some great additional info on general hurricane preparedness.

Option 43 in DHCP requires a very long hex string to be created (ours is 188 hex characters long). When adding this option to the Cisco ASA firewall via the ASDM (GUI) interface I get the following error message:

hex data is limited to 100 characters

I thought I was going to be in trouble, but it appears that there is no such limitation when adding this command via the command line interface (CLI) on the firewall via Telnet or SSH.

dhcpd option 43 hex 010cXXXXXXXXXX interface inside

So while I prefer to do most things through the ASDM when possible this is another case when Cisco hasn’t reached parity between the graphical and command line interfaces.

Test Configuration

The tests below use XenDesktop 5 SP1 and Provisioning Server 5.6 SP1.  The target VM is Windows 7 64-bit SP1 running on vSphere 4.1 and is configured with 4Gb RAM.

The vDisk is configured with 2Gb RAM cache.

This results in 4Gb total, 2Gb usable in Windows.

The Clown Car

We’ll begin with a fresh VDI session.  The PvS system tray applet shows 1% of the system cache is used.

We’ll then saturate the RAM cache by generating a 2Gb test file.

The test file is created successfully and the RAM cache is maxed at 100%.

Since we started with 30Mb and wrote another 2048Mb that should be it for our session, right?  Not quite.  Let’s copy a 1Gb Exchange ISO from the network…

…and download a 900Mb Windows service pack from the Internet.

“Shut the front door!” you say?  “Does Windows still work?”  Well, this video of the Swedish Chef plays perfectly.

I’m also able to create and save content in Paint, WordPad, etc.  Weird, right?  Let’s take a look at why this occurs.

The Explanation (per Citrix engineering)

When Citrix network stack interface driver (bnistack) MIoProcessWcRamWriteTransaction detects insufficient memory it returns successful to the caller (Windows) without actually saving the data to the RAM cache, so if Windows never reads the same data back, practically unlimited data can be added (though it may be limited by other factors like available disk space – Windows will prevent you from creating a 10Gb file when you have 8Gb free on your vDisk, for example).  Once the RAM cache becomes saturated errors are logged as follows:

Unfortunately by the time you attempt to access the Event Log your system is likely frozen, and if it’s a Standard image without redirected log files you’re out of luck when the frozen session is reset.

Digging Deeper

What does the explanation above really mean?  Does it hold up in practice?  Where does excess data go if it isn’t actually written?  What happens when you attempt to access excess data?  The examples below explore these questions.

Example 1

Upon logging on to a fresh VDI session and opening my test file creator, I have over 6 Gb free on the C drive…

…and memory usage is as follows:

I then create five separate 450Gb test files.

As the files are created RAM cache utilization increases, memory in use remains roughly constant, standby memory (Windows system cache) increases, and free memory decreases.

I can continue creating test files up to the limit of available hard drive space, at which point the following error is generated:

In other words, similar to “The Clown Car” above, I was able to generate test files that far exceeded the RAM cache before encountering an issue, and if I’d had a larger hard drive I would have been able to generate more.

So where did the data go?  We see in the table above that as test files are created the RAM cache increases in direct proportion to test file size until no more RAM cache remains.  Standby memory increases and free memory decreases in a similar manner, presumably showing that Windows is caching the most recently created data for quick access later, though this can’t really be considered “storage” for purposes of this analysis.

So what about the excess?  Let’s look at another example…

Example 2

Five 450 Gb test files are created, thus saturating RAM cache.  Attempting to open any of them results in an immediate system freeze and eventual reset.  In addition, deleting one or more of them then attempting to open one that remains results in an immediate freeze and eventual reset.  Finally, deleting all five files, creating a sixth, and attempting to open it results in an immediate freeze and eventual reset.  This suggests that once the RAM cache is saturated, attempting to access any content within it freezes the system.

Example 3

Four 450 Gb test files are created and RAM cache is at 89%.  Any one of them are opened.  A fifth is then created, thus saturating RAM cache.  We would appear to be in the same position as the previous example, but this time any of the first four files can be opened again and opening the fifth freezes the system.  This suggests that opening a file before the RAM cache is saturated somehow “protects” cache contents.

Example 4

Four 450 Gb test files are created and RAM cache is at 89%.  Any one of them are deleted.  RAM cache remains at 89%.  A fifth file is created.  RAM cache remains at 89%.  This suggests that deleted info isn’t reclaimed (or at least isn’t reported as reclaimed by the Virtual Disk Status applet) but is available for reuse when deleted.

Example 5

Four 450 Gb test files are created and RAM cache is at 89%.  Attempting to open all 4 maxes out memory in use, the system begins paging, the page file consumes additional RAM cache, and when RAM cache becomes saturated the system freezes.

Conclusions

The examples above suggest the following:

• RAM cache can be reused, but not reclaimed, up to the point of saturation.
• RAM cache saturation will likely not be apparent to end users.
• Once the saturation point is reached, data is at risk.
• If no files in the cache were accessed prior to saturation, any attempt to access cache contents will result in a freeze.
• If any file in the cache is accessed prior to saturation, contents in the cache prior to reaching saturation appear to be “protected.”
• Page file size and use is another important consideration.

Future

Citrix engineering has acknowledged that this is a bug, and has indicated that the fix should be to crash (BSOD) when RAM cache becomes saturated. “The fix should be to crash?  Really?” Yes, following the philosophy that existing user data should be protected and further corruption should be prevented.

Final Thoughts

As a best practice, a proof of concept with write cache on the server is recommended as it allows analysis of write cache size over time.  Once confidence is established with regard to write cache size, cost/benefit analysis can be performed relative to write cache location.

Thanks to my colleagues Matt Liebowitz and Niraj Patel for their help with this post.