Welcome to Kraft Kennedy

Today, Microsoft has released Service Pack 2 for Exchange 2010: http://blogs.technet.com/b/exchange/archive/2011/12/05/released-exchange-server-2010-sp2.aspx

While there are a number of hotfixes and other items included, there is also some key new functionality being introduced as well:

1. Cross-Site Silent Redirection for OWA – This is a tremendous improvement over current functionality where, if a firm leverages redirection between Client Access Servers in different sites (the preferred approach for optimal performance and implementation flexibility), users are prompted with a link and second authentication prompt if they login to a CAS server in a different site than where their mailbox is currently hosted.  With this new functionality, Exchange can be configured silently to redirect users to the correct CAS server in this situation (without reauthentication or prompting).
2. Address Book Policies – Address Book Policies provide the long-awaited native functionality support for what is typically referred to as GAL segmentation.  Previously, if firms wanted selectively to exclude some users or contacts from the GAL for specific subsets of users, the firm would need to create and manage security ACLs directly within ADSI.  This was unsupported in Exchange 2003 and 2010 and only narrowly supported for Exchange 2007 (with Dave Goldman’s specific whitepaper).  Address Book Policies will provide an object and policy based method for providing this functionality.
3. OWA Mini – This will be a lightweight, text-only version of OWA targeted for use on mobile devices or in low bandwidth/resolution scenarios.
4. Hybrid Configuration Wizard – This wizard will significantly reduce the number of steps required to streamline the process for establishing rich coexistence between an on-premises Exchange 2010 environment and Office 365 (formerly BPOS).

Due to the nature of the new features included, there is an Active Directory schema update required for SP2.

Recently, Microsoft added vSphere 5.0 as a supported hypervisor for Exchange 2010 to their Server Virtualization Validation Program support policy wizard.  Exchange 2010 RTM and SP1  are both listed as supported running on Windows 2008 RTM and Windows 2008 R2 on vSphere 5.0.  In addition, Exchange 2007 SP1, SP2 and SP3 are listed as supported on vSphere 5.0 as well but RTM is not.

Great news!

While most of our client deployments have gone quite smoothly from the perspective of stretching Exchange 2010 Database Availability Groups across multiple sites and WAN connectivities, I recently found myself troubleshooting an inconsistent issue at one client.  This environment’s topology was fairly straightforward, with two DAG members in one data center for local high availability and one DAG member in an alternate data center for remote site resiliency.  Creating the DAG, adding members, and adding mailbox database copies all presented no issues during the initial deployment although we did need to resolve some issues with database copy replication across the WAN.

As we approached our anticipated IT pre-pilot for the new Exchange 2010 environment, we started to notice significant issues in DAG communications across the WAN.  Specifically, we saw the following issues fairly consistently although, at some times, everything worked just fine:

• From the primary data center, viewing the mailbox database and associated copy status from the Exchange Management Console listed mount states for some databases as “Unknown” and copy status for all remote database copies as “ServiceDown.”  Running Get-MailboxDatabaseCopyStatus against the DAG member(s) in the remote data center reflected the same results.  Databases in an “Unknown” mount state corresponded to cases where the database was activated in one data center and status was being queried across the WAN from the other data center.
• Running “Get-DatabaseAvailabilityGroup -Status” would take an extremely long time to complete.
• Occasionally, databases would be listed in a dismounted state and, upon attempting to mount, an error message stating “Automount consensus not reached” would be returned and the mount would fail.
• Event logs on DAG members in both data centers would report sporadic occurrences of FailoverClustering events reporting that nodes in the repsective remote data center had been removed from cluster membership.
• Test-ReplicationHealth against DAG members across the WAN to the remote data center reported failures for ActiveManager (“Active Manager is in an unknown state”) and TasksRpcListener (“An error occurred while communicating with the Microsoft Exchange Replication service to test the health of the Tasks RPC Listener”).

The issue was clearly related to RPC requests traversing the WAN and having issues somewhere along the path from source to destination.  As a next step, I ran the “Validate a Configuration Wizard” for the DAG’s underlying Windows Failover Cluster and, sure enough, RPC errors were reported for queries that needed to cross the WAN to talk to cluster nodes not in the same data center as the node on which the wizard was run.  At this point, it was time to install Wireshark and run packet captures on either side of the WAN while executing Exchange actions or the cluster validation wizard to determine what was happening to the traffic.

Upon review of the packet captures, it was revealed that packets were being sent between DAG/cluster members that were larger than a standard 1500 byte packet and those packets were being fragmented in transit from source to destination.  Disabling various large TCP offload functionality of the NIC driver in use within the DAG/cluster members (vmxnet3 Ehternet Adapter) helped to bring the packet size down to 1500 bytes but the problems still occurred.  Running ping tests between the two data centers (ping -f -l <packet_size> hostname) revealed that the largest packet succeeding across the WAN was 1468 bytes.  Once the MTU of the NIC was reduced to match this value (via NETSH), everything began working perfectly.  The “Validate a Configuration Wizard” for the cluster completed without any unexpected warnings and all Exchange-related functionality was restored.

While disabling various functionality on the NIC and reducing the NIC’s MTU worked to solve the problem, it was certainly not ideal nor a long term solution for this environment.  Ultimately, determining where the issue lied in the WAN environment was key to identify how to resolve this issue without requiring non-standard configurations on various servers in the environment.  In working with the client’s networking team, it was understood that their particular WAN connectivity provided a Layer 2 Ethernet hand-off to each data center such that no router was in place on either side.  This explained why larger MTU packets were traversing the WAN and being fragmented in the process.  Coordination between the WAN provider and the remote data center’s networking team was required to determine where in the network path a device was unable to handle even a standard 1500 byte packet properly.

Ultimately, there were a few options for remediation at this client in the form of either obtaining jumbo frames support across their Layer 2 WAN or placing routers on either side of the WAN.  Both of these options would remove the requirement for non-standard configurations on the actual servers while still resolving the issue of communications between the data centers.

Finding Success with Exchange 2010, by David Carlson, was published by the International Legal Technology Assosication (ILTA) as part of the March 2011 white paper series.  If you are thinking about an Exchange 2010 project, give it a read.

The article contains guidance and discussion on the following topics:

• Planning Exchange 2010 deployment for law firms
• Features for law firms: Exchange 2010 Archiving, Litigation Holds and Search
• Large Mailboxes
• Database Availability Groups
• Load Balancing Exchange 2010 CAS Arrays
• Co-existence with Cloud Computing
• Regional Consolidation of Exchange
• Virtualization

I am thrilled to have been part of numerous successful Exchange 2010 deployments for law firms.  It is hard to believe that we have already migrated tens of thousands of Exchange mailboxes and I have enjoyed sharing this experience with many ILTA members.

I hope you enjoy reading the article and welcome hearing about your experience deploying Exchange 2010.  Please leave us a comment below!

David Carlson ILTA Kraft Kennedy KKL

This morning, my colleague Joe Hoegler pointed me to a new post on VMware’s Business Critical Applications blog entitled High Availability for Exchange 2010 without DAG.  Joe recently achieved his Microsoft Certified Master on Exchange 2010 and has a great deal of experience with Exchange.  He and I have worked together on projects where we’ve been successful in virtualizing Exchange 2010 on vSphere.  We both read the article and spent some time discussing it and both came to the same conclusions, so we wanted to share some of our collective thoughts.

Continue reading…

Over the last few months, there have been numerous reports of significant BES message delivery delays when running Exchange 2010, sometimes in excess of 20 minutes.  In addition to the primary symptom of message delivery delays to BES handhelds, environments experiencing this issue saw extremely high RPC Averaged Latency during periods of high BES utilization.  Despite being generously sized for RAM on Exchange 2010 Mailbox servers already, many firms dramatically increased RAM in their Mailbox servers to compensate for this issue, with amounts totaling 36 GB for 500 active BES users in many cases.  More information on some of the history and user experience with this issue can be found here.

While there were a number of technical reasons for this problem, a primary reason was changes to named properties in Exchange 2010.  To avoid legacy issues with finite numbers of named properties (see my previous blog post for some more information), functionality was changed in Exchange 2010 such that named properties are now stored per mailbox instead of per database and anonymous headers are not promoted so as to avoid issues with reaching the finite limits.  Since BES leverages a number of named properties for its own functionality, the latter caused some significant performance delays when BES attempted to query non-promoted named properties.

Microsoft and RIM have been working very closely together to remediate this issue and I’m happy to announce that some significant progress has been made.  If you are experiencing this problem currently, upgrading to MAPI/CDO 1.2.3 on your BES (see here) and BES itself to 5.0.2 Maintenance Release 4 (see here) should provide significant improvement on the BES functionality side.  Additional updates from Microsoft and RIM will be coming in the near future but, in the interim, these updates should help improve performance dramatically.

Update: The new MAPI/CDO was released as an updated version of 1.2.1, not version 1.2.3 as expected.  When downloading and installing the new MAPI/CDO, make sure you are installing version 1.2.1 dated 2/25/2011 and versioned 6.5.8211.0 (not the one dated 12/9/2009 and versioned 6.5.8147).  The link above will direct you to the correct version.

Recently, a client of ours with Exchange 2010 asked a question.   “How does Autonomy iManage’s Email Management work with Exchange 2010 Archive Mailboxes?”   A great question.     Kraft Kennedy has implemented the WorkSite Communication Server (WCS) and Email Filing Service (EFS) at several Exchange 2010 sites over the past year, but to this point, hadn’t come across a firm looking to leverage the built-in mail archiving functionality of Exchange 2010.   So we asked Autonomy how their EFS would handle it.  There was no mention of this functionality in any Release Notes, Install Guide, Admin Guide, or knowledge base tech note.   After being escalated to a product manager, we were informed that this functionality had not been part of any testing or QA for the 8.5 line of EFS.

Taking advantage of our Kraft Kennedy research lab, within a day I was able to connect our research iManage 8.5 system to our Exchange 2010 environment, and enabled the Archive Mailbox for my test account.     And for the first time in the history of mankind (I can only assume), tested the functionality.   Below you will find a brief summary of the environment and tests performed.
Continue reading…

I recently had the opportunity to attend Rotation 8 of the Microsoft Certified Master | Exchange 2010 program and am happy to report that I passed all three written exams and the final qualification lab exam on the first attempt.  I’m proud to be joining an elite community of only 19 MCMs worldwide prior to my rotation and those of my Rotation 8 colleagues that have passed or will after exam retakes.  I had the privilege of learning with some extremely talented individuals in my rotation and am looking forward to working with other MCMs in the near future.  I wanted to share some of my experiences in the hopes that it helps encourage others to pursue this elite certification.
Continue reading…

Kraft Kennedy is pleased to announce that Joe Hoegler, a Solution Architect based in their New York office, was recently awarded the Microsoft Certified Master certification on Exchange 2010.

Microsoft Certified Master (MCM) represents the highest technical certification that Microsoft offers and is awarded based upon successful completion of an intensive three week training course on-site at Microsoft’s headquarters along with three written exams and a final qualification lab exam.  Joe joins an elite group of  approximately 19 MCMs on Exchange 2010 worldwide to date. The Microsoft Certified Master program provides access to an exclusive community of other MCMs, members of the Exchange Server product group, as well as other valuable resources that can be leveraged at any time.

According to Microsoft, “a Microsoft Certified Master can help [an] organization assess its current messaging optimization level; map major initiatives and projects; rate the impact of identified gaps (value, cost, level of effort); and prioritize IT assets for maximum impact and return on investment.”

Joe has worked for Kraft Kennedy since 2004 and is a member of the Kraft Kennedy’s Infrastructure and Enterprise Systems practice group. He has advised numerous organizations on Exchange 2010 ranging from small firms to leading members of the AmLaw 100. Joe is also an active contributor to the Kraft Kennedy Technology Blog and has spoken at a number of ILTA events.

“When working with Joe, it is immediately clear that you have found someone with a deep understanding of the technology and also a keen awareness of the business requirements and best path to achieve them”, says David Carlson, Practice Leader at Kraft Kennedy. “This is a very exciting accomplishment and we are fortunate to have his talents as part of our team.”

You can read more about Joe and Exchange 2010 on the Kraft Kennedy Technology Blog.