While there are a number of hotfixes and other items included, there is also some key new functionality being introduced as well:

1. Cross-Site Silent Redirection for OWA – This is a tremendous improvement over current functionality where, if a firm leverages redirection between Client Access Servers in different sites (the preferred approach for optimal performance and implementation flexibility), users are prompted with a link and second authentication prompt if they login to a CAS server in a different site than where their mailbox is currently hosted.  With this new functionality, Exchange can be configured silently to redirect users to the correct CAS server in this situation (without reauthentication or prompting).
2. Address Book Policies – Address Book Policies provide the long-awaited native functionality support for what is typically referred to as GAL segmentation.  Previously, if firms wanted selectively to exclude some users or contacts from the GAL for specific subsets of users, the firm would need to create and manage security ACLs directly within ADSI.  This was unsupported in Exchange 2003 and 2010 and only narrowly supported for Exchange 2007 (with Dave Goldman’s specific whitepaper).  Address Book Policies will provide an object and policy based method for providing this functionality.
3. OWA Mini – This will be a lightweight, text-only version of OWA targeted for use on mobile devices or in low bandwidth/resolution scenarios.
4. Hybrid Configuration Wizard – This wizard will significantly reduce the number of steps required to streamline the process for establishing rich coexistence between an on-premises Exchange 2010 environment and Office 365 (formerly BPOS).

Due to the nature of the new features included, there is an Active Directory schema update required for SP2.

Great news!

As we approached our anticipated IT pre-pilot for the new Exchange 2010 environment, we started to notice significant issues in DAG communications across the WAN.  Specifically, we saw the following issues fairly consistently although, at some times, everything worked just fine:

• From the primary data center, viewing the mailbox database and associated copy status from the Exchange Management Console listed mount states for some databases as “Unknown” and copy status for all remote database copies as “ServiceDown.”  Running Get-MailboxDatabaseCopyStatus against the DAG member(s) in the remote data center reflected the same results.  Databases in an “Unknown” mount state corresponded to cases where the database was activated in one data center and status was being queried across the WAN from the other data center.
• Running “Get-DatabaseAvailabilityGroup -Status” would take an extremely long time to complete.
• Occasionally, databases would be listed in a dismounted state and, upon attempting to mount, an error message stating “Automount consensus not reached” would be returned and the mount would fail.
• Event logs on DAG members in both data centers would report sporadic occurrences of FailoverClustering events reporting that nodes in the repsective remote data center had been removed from cluster membership.
• Test-ReplicationHealth against DAG members across the WAN to the remote data center reported failures for ActiveManager (“Active Manager is in an unknown state”) and TasksRpcListener (“An error occurred while communicating with the Microsoft Exchange Replication service to test the health of the Tasks RPC Listener”).

The issue was clearly related to RPC requests traversing the WAN and having issues somewhere along the path from source to destination.  As a next step, I ran the “Validate a Configuration Wizard” for the DAG’s underlying Windows Failover Cluster and, sure enough, RPC errors were reported for queries that needed to cross the WAN to talk to cluster nodes not in the same data center as the node on which the wizard was run.  At this point, it was time to install Wireshark and run packet captures on either side of the WAN while executing Exchange actions or the cluster validation wizard to determine what was happening to the traffic.

Upon review of the packet captures, it was revealed that packets were being sent between DAG/cluster members that were larger than a standard 1500 byte packet and those packets were being fragmented in transit from source to destination.  Disabling various large TCP offload functionality of the NIC driver in use within the DAG/cluster members (vmxnet3 Ehternet Adapter) helped to bring the packet size down to 1500 bytes but the problems still occurred.  Running ping tests between the two data centers (ping -f -l <packet_size> hostname) revealed that the largest packet succeeding across the WAN was 1468 bytes.  Once the MTU of the NIC was reduced to match this value (via NETSH), everything began working perfectly.  The “Validate a Configuration Wizard” for the cluster completed without any unexpected warnings and all Exchange-related functionality was restored.

While disabling various functionality on the NIC and reducing the NIC’s MTU worked to solve the problem, it was certainly not ideal nor a long term solution for this environment.  Ultimately, determining where the issue lied in the WAN environment was key to identify how to resolve this issue without requiring non-standard configurations on various servers in the environment.  In working with the client’s networking team, it was understood that their particular WAN connectivity provided a Layer 2 Ethernet hand-off to each data center such that no router was in place on either side.  This explained why larger MTU packets were traversing the WAN and being fragmented in the process.  Coordination between the WAN provider and the remote data center’s networking team was required to determine where in the network path a device was unable to handle even a standard 1500 byte packet properly.

Ultimately, there were a few options for remediation at this client in the form of either obtaining jumbo frames support across their Layer 2 WAN or placing routers on either side of the WAN.  Both of these options would remove the requirement for non-standard configurations on the actual servers while still resolving the issue of communications between the data centers.

While there were a number of technical reasons for this problem, a primary reason was changes to named properties in Exchange 2010.  To avoid legacy issues with finite numbers of named properties (see my previous blog post for some more information), functionality was changed in Exchange 2010 such that named properties are now stored per mailbox instead of per database and anonymous headers are not promoted so as to avoid issues with reaching the finite limits.  Since BES leverages a number of named properties for its own functionality, the latter caused some significant performance delays when BES attempted to query non-promoted named properties.

Microsoft and RIM have been working very closely together to remediate this issue and I’m happy to announce that some significant progress has been made.  If you are experiencing this problem currently, upgrading to MAPI/CDO 1.2.3 on your BES (see here) and BES itself to 5.0.2 Maintenance Release 4 (see here) should provide significant improvement on the BES functionality side.  Additional updates from Microsoft and RIM will be coming in the near future but, in the interim, these updates should help improve performance dramatically.

Update: The new MAPI/CDO was released as an updated version of 1.2.1, not version 1.2.3 as expected.  When downloading and installing the new MAPI/CDO, make sure you are installing version 1.2.1 dated 2/25/2011 and versioned 6.5.8211.0 (not the one dated 12/9/2009 and versioned 6.5.8147).  The link above will direct you to the correct version.

My journey technically began in early 2010, when I discussed the possibility of attending MCM training with my firm’s management.  Later in 2010, I was informed that my firm would like to send me to this training and I began the application process.  Acceptance into the MCM program as a candidate occurs only after you apply and provide a few key documents for review by the program’s team.  Specifically, you must hold an MCITP: Enterprise Messaging Administrator certification and provide a current resume or CV, an outline of a recent project describing your specific roles and responsibilities, and a technical architecture document that you’ve written for a recent project.  You may be asked to participate in a phone interview to clarify any of your materials, although I was not.  Once all required materials have been submitted, you wait to hear back from the program team regarding their decision.

Once accepted, you must choose a three week rotation date (all are on-site in Redmond at Microsoft’s headquarters) and pay the rotation training fee ($18,500 as of the writing of this article).  As you approach your rotation date, you are provided with a substantial pre-reading list that includes a large portion of Exchange TechNet content and a number of recommended lab tasks to complete.  I strongly recommend that you build a lab environment and work through as many of these recommended lab tasks as you can.  They are extremely helpful in getting you familiar with concepts you may not have worked with previously (e.g. domain secure transport, cross-forest availability sharing, etc.) and will give you a head start on some labs that you may need to work through during your rotation.

My rotation began on January 17, 2011 and it included a diverse group of 22 individuals (plus an additional classmate completing some week 1 requirements from a previous rotation) from around the world.  Microsoft Premier Field Engineering and Microsoft Consulting Services had very strong representation in the rotation with approximately half of the candidates.

Week 1

Week 1 starts slow for the first couple of hours as everyone gets situated, receives an introduction to the program, and learns about the logistics for the next three weeks.  After that, it’s full-steam ahead for the entire week.  Class begins at 8:00am sharp each day and, on average, class ran until about 10:00pm each day during week 1.  In addition, a number of lab exercises were assigned throughout the course of each day and, as a result, most candidates remained in the lab well past midnight to complete all of the labs.  While you could certainly leave earlier, I wouldn’t recommend it since you would quickly fall behind on your labs if you did so.

Week 1 covers Client Access and Transport and is, in my opinion, the most difficult and intense content that is covered during the program.  You learn the deep technical details of AutoDiscover, proxying and redirection, namespace planning, RPC Proxy, secure Exchange publishing, Exchange routing, all phases of transport categorization, transport high availability, secured transport, and much more from two excellent instructors.  There is a ton of content to learn but I can honestly say that our instructors did an amazing job of reviewing all of the key details so that I came out with a clear understanding.  Discussing RPC Proxy concepts through analogies to elephants and shadow redundancy with paper plates certainly made for a memorable learning experience.

I spent my entire weekend studying for the week 1 written exam, proctored on the second Monday.  I split my weekend between studying all classroom content and my copious notes on Saturday and then in the lab reviewing key concepts with my fellow candidates on Sunday.  That strategy worked well for me but I know others in my class spent more and less time in the lab.  My rotation was very collaborative and working together in the lab was a great learning experience.  However, if you study better on your own, you may want to tailor your own schedule accordingly.

In total, I believe we reviewed about 1,000 slides of content and spent over 110 hours in the classroom in week 1.

Week 2

Week 2 began with our week 1 written exam, bright and early at 8:00am sharp on Monday.  The format is similar to most other MCP exams but the content is, as expected, significantly more difficult.  No matter how well you feel you understand the content of week 1, do not underestimate the difficultly of this exam.  The instructors are trying to gauge if you truly understand the underlying content and, as a result, this exam is extremely difficult.  According to many, the week 1 exam is easily the hardest of the three written exams due to both the difficulty and sheer amount of associated content.  If you don’t pass on the first try, don’t be discouraged.  Brush yourself off, prepare for week 2’s training, and worry about retaking the exam later.

Week 2 covers Mailbox and Database Internals, Storage, Database Availability Groups, and High Availability.  Again, the quality of material and instructors was outstanding.  While instruction didn’t go as late as week 1 (although we had instruction until about 9:30pm on the Monday of our week 1 exam), the amount of content covered was about the same.  Paper plates made a repeat appearance when reviewing Datacenter Activation Coordination concepts and, in general, I thoroughly enjoyed the content.  In week 2, you will learn the deep technical details of ESE, Jet, retention and archiving, performance and scalability validation, Mailbox Server role sizing, storage design and sizing, storage validation, and everything you could possibly want to know about DAGs and HA.

I took a similar approach to studying on the weekend (slide and note review plus collaborative studying in the lab) but spent a little more time in the lab than for week 1.  Again, it was a great experience to study with my fellow candidates.

In total, we reviewed about 1,000 slides and spent about 60 hours in the classroom for instruction and probably another 30-40 hours studying.

Week 3

Week 3 began with our week 2 written exam, again at 8:00am sharp on Monday.  The format is the same as week 1 although, again, much more difficult than any other Microsoft exam you’ve taken before.  Once the exam is over, you can take a quick break and then dive right back into content.

Week 3 covers a number of diverse topics and is designed to tie a number of concepts together.  In week 3, you’ll cover Unified Messaging, Operations, Virtualization, RBAC, Migration and Coexistence, Load Balancing, Federation, and Multi-Forest Deployments.  A good portion of the week 3 content is consultative in nature, specifically that around migration and coexistence and multi-forest deployments.  Many of the instructors for week 3 are Solution Architects at Microsoft, who focus heavily on the consultative side of Exchange.

Due to the course’s timing, you don’t have your third weekend for studying.  The week 3 written exam was held on Friday afternoon and, while you are taking the exam, your lab environment is being rebuilt in preparation for the final qualification lab exam, which is proctored on Saturday.  As a result, you need to be very careful with budgeting your in-week studying time during week 3.  You need to prepare for the week 3 written exam but, simultaneously, you need to be preparing for the qualification lab as well.

Qualification Lab

The qualification lab is the culmination of your three weeks of training and was a truly humbling experience.  While I cannot divulge a lot of information about the exam due to NDA restrictions on MCM candidates, the basic format is that you are given a number of seemingly simple tasks to complete but that have a number of items broken that prevent those tasks from completing normally.  Your goal is to fix as many tasks as possible in the 6 hours allotted to you for the lab.  The exam is not designed to be completely finished, so don’t be overwhelmed when you start working on it.  Given that, don’t spend too much time on any single task.  If you find yourself getting stuck, switch to another task and come back later.

Many of the instructors recommended working collaboratively with fellow candidates to prepare for the qualification lab and I couldn’t agree with that more.  Each candidate has had a different set of experiences that led him or her to this course and each may have seen or troubleshot different types of issues.  Pooling your shared knowledge and helping each other not only increases your chances of passing the qualification lab but also helps strengthen your understanding of topics on which you feel confident.

Once the qualification lab is over, take the time to unwind and celebrate with your class.  My class and I arranged an excellent steak dinner at Metropolitan Grill in Seattle and had a great time celebrating the completion of our rotation.  We even saw Bill and Melinda Gates at a table right outside of our private room!

You must wait about a week to learn the results of your qualification lab but, when you finally receive that email with your results, it is truly an amazing feeling.  It is great to see all of your hard work and dedication translate into something that not many other people in the world have been able to accomplish.

Final Thoughts

Overall, this program is truly an amazing experience and one that I strongly recommend pursuing if you have interest and are able.  However, make no mistake that this is an extremely difficult course and requires a tremendous amount of personal sacrifice.  Being away from my wife for three straight weeks and, on top of that, having limited time each night to talk on the phone due to the sheer volume of work and studying in addition to the 3 hour time difference was extremely difficult.  I am very grateful to have such a supportive and encouraging wife, family, and friends to help me through the course.

To give yourself the best chance for success, complete each and every lab exercise assigned to you, including the optional ones.  While no one will be checking that you’ve completed them, anything that is covered in class is a potential topic for written exams and the qualification lab.  While it involves a lot of effort, completing all of the lab exercises will put you in the best position to succeed in the course.

Get to know your fellow candidates.  Each rotation is a diverse group of extremely intelligent and talented individuals, each with different experiences.  I am honored to have worked with such great people in my rotation and am looking forward to staying in touch for years to come.

For more information about the MCM program, please visit http://www.microsoft.com/learning/en/us/certification/master.aspx.  I’m also happy to answer any questions that I can but please understand that I may not be able to disclose some details.

The issue between BES and Exchange 2010 SP1 is that, prior to SP1, RIM’s published guidance included the requirement that you increase the number of allowed connections to the Address Book service via the MaxSessionsPerUser key in the microsoft.exchange.addressbook.service.exe.config file.  This key was deprecated with SP1.  As a result, RIM has published new guidance as of September 13, 2010 regarding how to properly configure Exchange 2010 SP1 to support BES.  Elan Shudnow previously published a blog post about this issue and RIM has since updated their documentation to reflect the changes required for client throttling.

In short, configuring the custom Throttling Policy for your BES service account (see my post here for more information on Client Throttling in Exchange 2010) via the cmdlets below will allow BES to work properly.  Again, please note RIM’s current supportability statement before upgrading production environments.

New-ThrottlingPolicy BESPolicy -RCAMaxConcurrency $null -EWSFindCountLimit $null -EWSMaxConcurrency $null
Set-Mailbox “BESAdmin” -ThrottlingPolicy BESPolicy

While modifying the FQDN of the SMTP Virtual Server certainly does mask this, the actual internal IP address of the server is still listed in the e-mail message headers and, as such, the identity of the server isn’t actually masked.  Furthermore, if a hacker were to compromise a perimeter firewall, not knowing the exact name of the server providing SMTP services will not deter much.

In most client environments, modifying this FQDN is mostly harmless, provided that the configured FQDN is resolvable in internal DNS.  This is because, when Exchange routing calculates a next hop, the announced FQDN of the appropriate SMTP Virtual Server is used.  If the configured name was not resolvable, Exchange would queue and bounce messages destined for a specific server if that server was the designated next hop for SMTP routing.  In addition to ensuring the configured FQDN is resolvable, it is also important to understand exactly why the change was made and the ramifications to doing so.

In one client environment I worked on, a specific Exchange 2003 server (SERVER2003_TLS, for example) was defined as the server to use for outbound communications to specific remote domains that required TLS (via an SMTP Connector and being defined as the associated bridgehead).  SERVER2003_TLS also had its SMTP Virtual Server FQDN configured to something along the lines of mail.client.com instead of the actual server FQDN.  mail.client.com was also the external and internal base URL for accessing OWA and ActiveSync services.

Everything worked fine in the Exchange 2003 environment because mail.client.com pointed to SERVER2003_TLS in DNS.  However, when Exchange 2010 was introduced and mail.client.com was moved to the client’s new Exchange 2010 load balanced CAS Array for coexistence purposes, TLS mail started bouncing with a “local loop detected” NDR.

The reason for the NDR was that a message sent to one of the remote domains defined on the SMTP Connector that required TLS would query SERVER2003_TLS since it was defined as the bridgehead for that SMTP Connector.  Since SERVER2003_TLS’s SMTP Virtual Server responded as mail.client.com, mail.client.com would be used for next hop routing.  However, since mail.client.com had been repointed to the Exchange 2010 CAS Array for CAS coexistence and these servers were also hosting the HTS role, these servers accepted the messages.  Since Exchange 2010 had not yet been configured with a direct outbound SMTP path to the Internet, all outbound Internet e-mail was routed across the coexistence Routing Group Connectors to one of two servers defined as transport servers for coexistence mail flow.

These Exchange 2003 servers then identified the same SMTP Connector for the configured remote domains requiring TLS and SERVER2003_TLS as the appropriate bridgehead, queried SERVER2003_TLS’s SMTP Virtual Server, received mail.client.com as the next hop, and the cycle continued until the message reached the maximum hop count limit.  Any number of solutions could have been used to resolve this issue once it was identified but changing the SMTP Virtual Server on SERVER2003_TLS to something other than mail.client.com was the preferred approach.

While this specific issue may not happen elsewhere, it highlights the need to truly understand the nature of changes made that alter default functionality.  In this case, changing the FQDN of the SMTP Virtual Server didn’t provide a lot of benefit to begin with but resulted in major mail flow issues down the road.

For more in my series on Exchange 2010 Notes from the Field, please click here.

By default in Exchange 2010, simultaneous RPC connections per user are limited to 20 (via the RCAMaxConcurrency parameter of the default client throttling policy).  In cases where many users share mailbox resources via delegate access, such as with attorney/secretary pairings, it is possible to reach the default limit of 20 simultaneous RPC connections for a mailbox.  Furthermore, the legacy Outlook 2003 client didn’t release any RPC connections associated with opening shared calendars until the Outlook client was closed.  If the RPC concurrency limit is reached, the affected user will receive an “unable to open your default e-mail folders” type error message when launching Outlook.

Many third party integrated applications require many simultaneous connections to Exchange via a defined service account due to how the application must interact with Exchange.  BlackBerry Enterprise Server and InterAction are two such applications that make many MAPI requests to Exchange through their service accounts and will quickly reach the defined limit of 20 simultaneous RPC connections.  Usually, a vendor’s documentation will dictate if the default Exchange 2010 client throttling policy must be modified to support their application but, in some cases, this is not the case.

Since client throttling is a good thing, it is not recommended to simply set any of the parameters associated with the default policy to no limit.  Instead, I recommend increasing the RCAMaxConcurrency limit on the default policy to a more reasonable value, such as 32 or 40.  For third party applications, I recommend creating a custom client throttling policy that either sets RCAMaxConcurrency to no limit or to a value that the vendor requires.  This custom policy can then be applied to the service accounts used by the third party applications.  Please note that, to set RCAMaxConcurrency to no limit, the parameter should be set to $null.  Setting the value to 0 will effectively prevent any MAPI access for the users to which the client throttling policy is applied.

For more in my series on Exchange 2010 Notes from the Field, please click here.

The store driver couldnt deliver the public folder replication message “Hierarchy (PublicFolder@client.com)” because the following error occurred: The Active Directory user wasn’t found.

These errors likely coincide with some public folder replication issues but, more importantly, can also result in NDRs when messages are sent to mail-enabled public folders!  This issue occurs because, even if you properly decommission Exchange 2003, the Servers containers within your legacy Exchange 2003 Administrative Groups still exist within Active Directory, albeit empty.  Exchange assumes that, if a Servers container exists (even if it is empty), a System Attendant object will also exist somewhere inside of it but, if all of your Exchange 2003 servers have been decommissioned, those System Attendant objects actually do not exist.

This issue has been recognized as a bug within Exchange (see the MS Exchange Team Blog article here) and a fix a scheduled for an upcoming Update Rollup release (perhaps Update Rollup 5).  In the meantime, you can safely delete these empty Servers containers via ADSI Edit but make sure that these containers completely empty before doing so.

For more in my series on Exchange 2010 Notes from the Field, please click here.

To resolve this issue for the specific case at my client, we simply needed to enable inheritance on the OUs or users where it had previously been disabled.

Resolving this issue for members of privileged groups is a bit more complicated.  Basically, the lack of inheritance is by design for users that are members of privileged AD groups.  Every hour, a background process runs on domain controllers to apply the permissions assigned to the AdminSDHolder template object to all members of privileged groups.  You can review the permissions that will be applied by launching Active Directory Users and Computers, enabling Advanced Features within the View menu, and then reviewing the security permissions of the AdminSDHolder object within the System OU.

The true solution is to provide administrators with separate administrative-only accounts (e.g. JohnAdmin.admin) that are members of the required AD groups and have these administrators use normal, non-privileged accounts (e.g. JohnAdmin) for e-mail functionality.  In some environments, this may not be possible and, as a result, you have two workarounds.  First, you could modify the permissions on the AdminSDHolder template object to include the required Exchange permissions.  I don’t recommend this since you would be modifying a fairly important and engrained aspect of Active Directory for what should be a few isolated users.  Instead, you could temporarily enable inheritance on your administrative users and, as long as you configure these users’ ActiveSync devices before the next application of AdminSDHolder permissions, it will work just fine.  Once an ActiveSync device is provisioned for the user, these special Exchange permissions are no longer required.

For more information on AdminSDHolder, the associated default permissions, and instructions for modifying these permissions, please refer to http://policelli.com/blog/?p=136.

For more in my series on Exchange 2010 Notes from the Field, please click here.