When: Friday, February 24, 2012
Time: 12:00-1:00 Eastern
Register: Click Here to Register

Presenter BIO:

Joe Hoegler is the Practice Leader of Kraft Kennedy’s Infrastructure and Enterprise Systems Practice Group. He provides technical leadership and strategic guidance on client engagements involving a broad range of law firm technologies and is responsible for directing technology strategy and providing technical management at the firm. As of December 2011, he has led or advised over 35 law firm clients totaling over 35,000 users on projects related to Exchange 2010, ranging in size from 30 to 6,000 users. Joe is a Microsoft Certified Master on Exchange 2010, one of only approximately 50 people worldwide who hold this certification.

The first step is to create a contact that will serve as the recipient for the SMS message.  You will need to consult the settings for your particular carrier to find the format for this address.  For AT&T users, the format will be [10-digit phone number]@txt.att.net.  Once the contact is created, create an email rule similar to the one below:

This rule will tell Exchange to forward any message marked as “High Importance” to the contact that was created for the SMS address. While the above view is from a Windows Outlook client, the same can be accomplished through OWA, or in Entourage or Outlook 2011 for Mac.

Within a few minutes (depending on your reception and your carrier), you will receive a text version of the email message that was marked with high importance.  Because of the inherent 160 character limitation of SMS messaging, it is unlikely that you will be able to read the entire message, however the SMS message will alert you to the fact that you have a priority email and you can then read it in the native iOS email client. Keep in mind, the source of the text message may vary based upon your carrier. Here is how the text message may appear on an AT&T connected device. Note the highlighted portion of the text.

One thing to be careful of… Normal text messaging rates will apply, so if you receive a large number of priority messages, and do not have an unlimited messaging plan, this could get expensive very quickly.  You can lower the number of messages that get forwarded by tweaking the rule to filter for certain senders, but its always a good idea to be aware of your SMS plan.

As we approached our anticipated IT pre-pilot for the new Exchange 2010 environment, we started to notice significant issues in DAG communications across the WAN.  Specifically, we saw the following issues fairly consistently although, at some times, everything worked just fine:

• From the primary data center, viewing the mailbox database and associated copy status from the Exchange Management Console listed mount states for some databases as “Unknown” and copy status for all remote database copies as “ServiceDown.”  Running Get-MailboxDatabaseCopyStatus against the DAG member(s) in the remote data center reflected the same results.  Databases in an “Unknown” mount state corresponded to cases where the database was activated in one data center and status was being queried across the WAN from the other data center.
• Running “Get-DatabaseAvailabilityGroup -Status” would take an extremely long time to complete.
• Occasionally, databases would be listed in a dismounted state and, upon attempting to mount, an error message stating “Automount consensus not reached” would be returned and the mount would fail.
• Event logs on DAG members in both data centers would report sporadic occurrences of FailoverClustering events reporting that nodes in the repsective remote data center had been removed from cluster membership.
• Test-ReplicationHealth against DAG members across the WAN to the remote data center reported failures for ActiveManager (“Active Manager is in an unknown state”) and TasksRpcListener (“An error occurred while communicating with the Microsoft Exchange Replication service to test the health of the Tasks RPC Listener”).

The issue was clearly related to RPC requests traversing the WAN and having issues somewhere along the path from source to destination.  As a next step, I ran the “Validate a Configuration Wizard” for the DAG’s underlying Windows Failover Cluster and, sure enough, RPC errors were reported for queries that needed to cross the WAN to talk to cluster nodes not in the same data center as the node on which the wizard was run.  At this point, it was time to install Wireshark and run packet captures on either side of the WAN while executing Exchange actions or the cluster validation wizard to determine what was happening to the traffic.

Upon review of the packet captures, it was revealed that packets were being sent between DAG/cluster members that were larger than a standard 1500 byte packet and those packets were being fragmented in transit from source to destination.  Disabling various large TCP offload functionality of the NIC driver in use within the DAG/cluster members (vmxnet3 Ehternet Adapter) helped to bring the packet size down to 1500 bytes but the problems still occurred.  Running ping tests between the two data centers (ping -f -l <packet_size> hostname) revealed that the largest packet succeeding across the WAN was 1468 bytes.  Once the MTU of the NIC was reduced to match this value (via NETSH), everything began working perfectly.  The “Validate a Configuration Wizard” for the cluster completed without any unexpected warnings and all Exchange-related functionality was restored.

While disabling various functionality on the NIC and reducing the NIC’s MTU worked to solve the problem, it was certainly not ideal nor a long term solution for this environment.  Ultimately, determining where the issue lied in the WAN environment was key to identify how to resolve this issue without requiring non-standard configurations on various servers in the environment.  In working with the client’s networking team, it was understood that their particular WAN connectivity provided a Layer 2 Ethernet hand-off to each data center such that no router was in place on either side.  This explained why larger MTU packets were traversing the WAN and being fragmented in the process.  Coordination between the WAN provider and the remote data center’s networking team was required to determine where in the network path a device was unable to handle even a standard 1500 byte packet properly.

Ultimately, there were a few options for remediation at this client in the form of either obtaining jumbo frames support across their Layer 2 WAN or placing routers on either side of the WAN.  Both of these options would remove the requirement for non-standard configurations on the actual servers while still resolving the issue of communications between the data centers.

I recommend reading Andre’s post to get more information on this topic.  The short version is this – Exchange/Outlook best practices do not necessarily work in a VDI environment.

When using Outlook in an Exchange environment, it is recommended to use Cached Exchange Mode.  In this mode a copy of the user’s mailbox is downloaded into an OST file and stored offline on the user’s desktop.  This mode offers better performance for the end user and reduces utilization on the Exchange environment as well.  In addition, using Cached Exchange Mode allows users to use Outlook Instant Search for fast searching of items in their mailbox.  Instant Search works by indexing the contents of the OST file so all searches occur locally and not on the Exchange server, further improving performance and reducing utilization on Exchange.

VDI environments that we see at our clients are typically configured as non-persistent or floating pools of desktops.  That is, each user connects to a pool of identical desktops and grabs whatever desktop is available.  When the user logs off, any changes written to the VDI desktop are discarded and the desktop returns to a pristine state.  There are mechanisms and tools in place to make sure user data is retained at logoff.

So if user data is retained at logoff, why can’t we use Cached Exchange Mode in non-persistent VDI environments?

• The OST file is equal in size to the user’s mailbox so storing a 15-30GB OST (not unusual at our clients) is not that practical from a performance or storage perspective.  If this data is being stored on the SAN, then you’re essentially doubling your Exchange storage (which may already be doubled or tripled if you’re using Exchange 2010 w/ DAGs).  In addition, the length of time it would take to download that file every time and the I/O impact that would cause makes it completely impractical.
• OST files are not supported when stored on network shares, so redirecting the OST to a home directory is out.
• Indexing of files on virtual desktops is typically disabled to reduce I/O demands.  This would prevent the use of Outlook Instant Search even if the OST was present.

For these and other reasons, Outlook is typically configured in Online mode when used with VDI.  This keeps all mailbox operations and searches on the Exchange server, placing the processing and I/O burden solely on the Exchange environment.  That sounds bad, but advances in Exchange technology specifically with Exchange 2010 have made this much less of an issue.  In fact, Microsoft states that IOPS requirements for Cached Exchange Mode and Online mode are essentially equal now, meaning there is no I/O “penalty” for using Online mode.

Exchange 2010 Performance

Let’s take an example scenario of 500 VDI users all running Outlook in Online mode against an Exchange 2010 backend.  We’ll estimate high and assume they all have a mailbox profile of 300 messages sent/received per day.  According to Microsoft that amounts to a 0.3 IOPS per user requirement of 150 IOPS total, or roughly equivalent to the capabilities of one 15k RPM disk.

We can make it worse and assume that all 500 users also have Blackberry devices, which introduces a multiplier of 2.16 IOPS per user.  (Note: this number is specific to Exchange 2007 as I haven’t been able to find a definitive number for Exchange 2010, but expect that the multipler will be even lower)  That brings our IOPS per user calculation to 0.65 (2.16 x 0.3), bringing the total IOPS requirement for all 500 users to 324 IOPS or roughly three 15K RPM disks (excluding RAID penalty, though this is less of an issue on modern storage arrays).  It is unlikely that all 500 users would have Blackberry devices and also send/receive 300 messages/day, so the actual requirement is likely lower than that.

Searching in Outlook

What about searching from Outlook?  Doesn’t that impose a significant IOPS burden on the Exchange server?  According to Microsoft, the penalty for searching in Online mode is just 10-15% of the database I/O based on user profile.  Using our 300 messages sent/receive per day number of 0.3 IOPS/user, 15% of 0.3 is just 0.045 additional IOPS. It’s no wonder that Microsoft states “Search catalog read I/O occurs when clients issue search queries, and it’s a rare enough occurrence to not be relevant to Exchange 2010 storage design.”

In other words – it doesn’t impact the Exchange environment enough to matter.  Is searching in Online mode slower than using Outlook Instant Search?  No question it’s slower, but search performance has improved significantly with Exchange 2010 so it isn’t as bad as it was in the past.  And the impact on the performance of the Exchange environment is almost negligible.

Summary

Quick breakdown of the numbers used above:

500 users each with a Blackberry and sending/receiving 300 messages/day = 324 IOPS

Impact of all 500 users performing searches = 0.3 x 15% x 500 = 22.5 additional IOPS

Drawbacks to Online Mode

It’s not all rainbows and unicorns when it comes to using Outlook in Online mode unfortunately.  In my opinion the major drawback of using Outlook in Online Mode is not performance but rather availability.

In Exchange 2010, the RPC endpoint has been moved from the Mailbox server to the Client Access Server.  This change means, among other things, that users are not interrupted when moving databases between nodes in a DAG.  Unfortunately that is only true if the user is using Cached Exchange Mode, as Online Mode users experience a brief period where Outlook becomes frozen and unresponsive.

Similarly, if the Exchange environment experiences an outage then Online Mode users are completely frozen and locked out of their mailbox with no access to messages.  Cached Exchange Mode users would not be able to send and receive new messages but would still have access to the contents of their mailbox since they are working off a locally cached copy.

Conclusion

There are many significant advances in Exchange 2010 that have reduced the overall disk I/O requirements down to a tiny fraction of what was required in previous versions.  There is no longer any penalty for using Online Mode vs. Cached Exchange Mode in terms of IOPS required per user, and the example above clearly shows that it is easy to meet the I/O demands of even heavy Exchange users.  The user experience may be slightly better with Cached Exchange Mode but likely not by much.

Most of the issues with Outlook in a VDI environment are around availability, not performance.  Although these cannot be eliminated, properly architecting your Exchange 2010 environment can help eliminate single points of failure and provide excellent availability to users.

If you’re considering VDI and are concerned about Outlook performance, I’d strongly recommend moving to Exchange 2010.  Many of the problems are addressed in Exchange 2010 and it can deliver a good Outlook experience for all VDI users.

The requirements to participate in the Microsoft Partner Program have recently evolved to help differentiate technical and business capabilities among participants; Kraft Kennedy has risen to the challenge by quickly exceeding the goals set forth by the program.

Each competency requires specific individuals with deep technical skills, Microsoft verified customer references, and challenging certification exams to be completed.  This commitment demonstrates our breadth, deep specialization, and proven expertise across a range of Microsoft technologies.

About Kraft Kennedy

Kraft Kennedy provides business and technology-related consulting services to the legal community. By combining outstanding technical skills with an intimate knowledge of our clients’ business and information needs we tailor solutions that enhance attorney productivity, effectiveness, and client value.

We focus on the business needs of the client and ensure that technology is used to enhance, not inhibit their business. KK’s talented staff of strategic consultants, project managers, and network consultants have years of experience with hundreds of projects for firms from small to large. Our services portfolio includes advanced infrastructure projects, business continuity and data center consolidation, desktop deployment, network design and implementation, storage design and replication, and messaging systems migration among others. Our Microsoft specialties include: Desktop, Server Platform, Unified Communications, Portals and Collaboration, Search, Systems Management, Virtualization, and Small Business Specialist Community.

While modifying the FQDN of the SMTP Virtual Server certainly does mask this, the actual internal IP address of the server is still listed in the e-mail message headers and, as such, the identity of the server isn’t actually masked.  Furthermore, if a hacker were to compromise a perimeter firewall, not knowing the exact name of the server providing SMTP services will not deter much.

In most client environments, modifying this FQDN is mostly harmless, provided that the configured FQDN is resolvable in internal DNS.  This is because, when Exchange routing calculates a next hop, the announced FQDN of the appropriate SMTP Virtual Server is used.  If the configured name was not resolvable, Exchange would queue and bounce messages destined for a specific server if that server was the designated next hop for SMTP routing.  In addition to ensuring the configured FQDN is resolvable, it is also important to understand exactly why the change was made and the ramifications to doing so.

In one client environment I worked on, a specific Exchange 2003 server (SERVER2003_TLS, for example) was defined as the server to use for outbound communications to specific remote domains that required TLS (via an SMTP Connector and being defined as the associated bridgehead).  SERVER2003_TLS also had its SMTP Virtual Server FQDN configured to something along the lines of mail.client.com instead of the actual server FQDN.  mail.client.com was also the external and internal base URL for accessing OWA and ActiveSync services.

Everything worked fine in the Exchange 2003 environment because mail.client.com pointed to SERVER2003_TLS in DNS.  However, when Exchange 2010 was introduced and mail.client.com was moved to the client’s new Exchange 2010 load balanced CAS Array for coexistence purposes, TLS mail started bouncing with a “local loop detected” NDR.

The reason for the NDR was that a message sent to one of the remote domains defined on the SMTP Connector that required TLS would query SERVER2003_TLS since it was defined as the bridgehead for that SMTP Connector.  Since SERVER2003_TLS’s SMTP Virtual Server responded as mail.client.com, mail.client.com would be used for next hop routing.  However, since mail.client.com had been repointed to the Exchange 2010 CAS Array for CAS coexistence and these servers were also hosting the HTS role, these servers accepted the messages.  Since Exchange 2010 had not yet been configured with a direct outbound SMTP path to the Internet, all outbound Internet e-mail was routed across the coexistence Routing Group Connectors to one of two servers defined as transport servers for coexistence mail flow.

These Exchange 2003 servers then identified the same SMTP Connector for the configured remote domains requiring TLS and SERVER2003_TLS as the appropriate bridgehead, queried SERVER2003_TLS’s SMTP Virtual Server, received mail.client.com as the next hop, and the cycle continued until the message reached the maximum hop count limit.  Any number of solutions could have been used to resolve this issue once it was identified but changing the SMTP Virtual Server on SERVER2003_TLS to something other than mail.client.com was the preferred approach.

While this specific issue may not happen elsewhere, it highlights the need to truly understand the nature of changes made that alter default functionality.  In this case, changing the FQDN of the SMTP Virtual Server didn’t provide a lot of benefit to begin with but resulted in major mail flow issues down the road.

For more in my series on Exchange 2010 Notes from the Field, please click here.

By default in Exchange 2010, simultaneous RPC connections per user are limited to 20 (via the RCAMaxConcurrency parameter of the default client throttling policy).  In cases where many users share mailbox resources via delegate access, such as with attorney/secretary pairings, it is possible to reach the default limit of 20 simultaneous RPC connections for a mailbox.  Furthermore, the legacy Outlook 2003 client didn’t release any RPC connections associated with opening shared calendars until the Outlook client was closed.  If the RPC concurrency limit is reached, the affected user will receive an “unable to open your default e-mail folders” type error message when launching Outlook.

Many third party integrated applications require many simultaneous connections to Exchange via a defined service account due to how the application must interact with Exchange.  BlackBerry Enterprise Server and InterAction are two such applications that make many MAPI requests to Exchange through their service accounts and will quickly reach the defined limit of 20 simultaneous RPC connections.  Usually, a vendor’s documentation will dictate if the default Exchange 2010 client throttling policy must be modified to support their application but, in some cases, this is not the case.

Since client throttling is a good thing, it is not recommended to simply set any of the parameters associated with the default policy to no limit.  Instead, I recommend increasing the RCAMaxConcurrency limit on the default policy to a more reasonable value, such as 32 or 40.  For third party applications, I recommend creating a custom client throttling policy that either sets RCAMaxConcurrency to no limit or to a value that the vendor requires.  This custom policy can then be applied to the service accounts used by the third party applications.  Please note that, to set RCAMaxConcurrency to no limit, the parameter should be set to $null.  Setting the value to 0 will effectively prevent any MAPI access for the users to which the client throttling policy is applied.

For more in my series on Exchange 2010 Notes from the Field, please click here.

The store driver couldnt deliver the public folder replication message “Hierarchy (PublicFolder@client.com)” because the following error occurred: The Active Directory user wasn’t found.

These errors likely coincide with some public folder replication issues but, more importantly, can also result in NDRs when messages are sent to mail-enabled public folders!  This issue occurs because, even if you properly decommission Exchange 2003, the Servers containers within your legacy Exchange 2003 Administrative Groups still exist within Active Directory, albeit empty.  Exchange assumes that, if a Servers container exists (even if it is empty), a System Attendant object will also exist somewhere inside of it but, if all of your Exchange 2003 servers have been decommissioned, those System Attendant objects actually do not exist.

This issue has been recognized as a bug within Exchange (see the MS Exchange Team Blog article here) and a fix a scheduled for an upcoming Update Rollup release (perhaps Update Rollup 5).  In the meantime, you can safely delete these empty Servers containers via ADSI Edit but make sure that these containers completely empty before doing so.

For more in my series on Exchange 2010 Notes from the Field, please click here.

To resolve this issue for the specific case at my client, we simply needed to enable inheritance on the OUs or users where it had previously been disabled.

Resolving this issue for members of privileged groups is a bit more complicated.  Basically, the lack of inheritance is by design for users that are members of privileged AD groups.  Every hour, a background process runs on domain controllers to apply the permissions assigned to the AdminSDHolder template object to all members of privileged groups.  You can review the permissions that will be applied by launching Active Directory Users and Computers, enabling Advanced Features within the View menu, and then reviewing the security permissions of the AdminSDHolder object within the System OU.

The true solution is to provide administrators with separate administrative-only accounts (e.g. JohnAdmin.admin) that are members of the required AD groups and have these administrators use normal, non-privileged accounts (e.g. JohnAdmin) for e-mail functionality.  In some environments, this may not be possible and, as a result, you have two workarounds.  First, you could modify the permissions on the AdminSDHolder template object to include the required Exchange permissions.  I don’t recommend this since you would be modifying a fairly important and engrained aspect of Active Directory for what should be a few isolated users.  Instead, you could temporarily enable inheritance on your administrative users and, as long as you configure these users’ ActiveSync devices before the next application of AdminSDHolder permissions, it will work just fine.  Once an ActiveSync device is provisioned for the user, these special Exchange permissions are no longer required.

For more information on AdminSDHolder, the associated default permissions, and instructions for modifying these permissions, please refer to http://policelli.com/blog/?p=136.

For more in my series on Exchange 2010 Notes from the Field, please click here.

The issue can be caused by search folder problems in the Exchange 2003 mailbox and similar issues were fixed in Exchange 2007 via Service Pack 2.  Microsoft describes ways to purge the disconnected Exchange 2003 mailbox at http://support.microsoft.com/default.aspx?scid=kb;EN-US;930363 but the workarounds involve either a number of manual steps or reducing your mailbox retention settings to 0 days and allowing normal Exchange online maintenance to purge the mailboxes.  The latter would also purge any legitimately deleted mailboxes that you may want to retain for some period of time so the safest workaround may be to just wait for your normal online maintenance procedures to purge the mailboxes for you (30 days by default).

For more in my series on Exchange 2010 Notes from the Field, please click here.