While there are a number of hotfixes and other items included, there is also some key new functionality being introduced as well:

1. Cross-Site Silent Redirection for OWA – This is a tremendous improvement over current functionality where, if a firm leverages redirection between Client Access Servers in different sites (the preferred approach for optimal performance and implementation flexibility), users are prompted with a link and second authentication prompt if they login to a CAS server in a different site than where their mailbox is currently hosted.  With this new functionality, Exchange can be configured silently to redirect users to the correct CAS server in this situation (without reauthentication or prompting).
2. Address Book Policies – Address Book Policies provide the long-awaited native functionality support for what is typically referred to as GAL segmentation.  Previously, if firms wanted selectively to exclude some users or contacts from the GAL for specific subsets of users, the firm would need to create and manage security ACLs directly within ADSI.  This was unsupported in Exchange 2003 and 2010 and only narrowly supported for Exchange 2007 (with Dave Goldman’s specific whitepaper).  Address Book Policies will provide an object and policy based method for providing this functionality.
3. OWA Mini – This will be a lightweight, text-only version of OWA targeted for use on mobile devices or in low bandwidth/resolution scenarios.
4. Hybrid Configuration Wizard – This wizard will significantly reduce the number of steps required to streamline the process for establishing rich coexistence between an on-premises Exchange 2010 environment and Office 365 (formerly BPOS).

Due to the nature of the new features included, there is an Active Directory schema update required for SP2.

Great news!

As we approached our anticipated IT pre-pilot for the new Exchange 2010 environment, we started to notice significant issues in DAG communications across the WAN.  Specifically, we saw the following issues fairly consistently although, at some times, everything worked just fine:

• From the primary data center, viewing the mailbox database and associated copy status from the Exchange Management Console listed mount states for some databases as “Unknown” and copy status for all remote database copies as “ServiceDown.”  Running Get-MailboxDatabaseCopyStatus against the DAG member(s) in the remote data center reflected the same results.  Databases in an “Unknown” mount state corresponded to cases where the database was activated in one data center and status was being queried across the WAN from the other data center.
• Running “Get-DatabaseAvailabilityGroup -Status” would take an extremely long time to complete.
• Occasionally, databases would be listed in a dismounted state and, upon attempting to mount, an error message stating “Automount consensus not reached” would be returned and the mount would fail.
• Event logs on DAG members in both data centers would report sporadic occurrences of FailoverClustering events reporting that nodes in the repsective remote data center had been removed from cluster membership.
• Test-ReplicationHealth against DAG members across the WAN to the remote data center reported failures for ActiveManager (“Active Manager is in an unknown state”) and TasksRpcListener (“An error occurred while communicating with the Microsoft Exchange Replication service to test the health of the Tasks RPC Listener”).

The issue was clearly related to RPC requests traversing the WAN and having issues somewhere along the path from source to destination.  As a next step, I ran the “Validate a Configuration Wizard” for the DAG’s underlying Windows Failover Cluster and, sure enough, RPC errors were reported for queries that needed to cross the WAN to talk to cluster nodes not in the same data center as the node on which the wizard was run.  At this point, it was time to install Wireshark and run packet captures on either side of the WAN while executing Exchange actions or the cluster validation wizard to determine what was happening to the traffic.

Upon review of the packet captures, it was revealed that packets were being sent between DAG/cluster members that were larger than a standard 1500 byte packet and those packets were being fragmented in transit from source to destination.  Disabling various large TCP offload functionality of the NIC driver in use within the DAG/cluster members (vmxnet3 Ehternet Adapter) helped to bring the packet size down to 1500 bytes but the problems still occurred.  Running ping tests between the two data centers (ping -f -l <packet_size> hostname) revealed that the largest packet succeeding across the WAN was 1468 bytes.  Once the MTU of the NIC was reduced to match this value (via NETSH), everything began working perfectly.  The “Validate a Configuration Wizard” for the cluster completed without any unexpected warnings and all Exchange-related functionality was restored.

While disabling various functionality on the NIC and reducing the NIC’s MTU worked to solve the problem, it was certainly not ideal nor a long term solution for this environment.  Ultimately, determining where the issue lied in the WAN environment was key to identify how to resolve this issue without requiring non-standard configurations on various servers in the environment.  In working with the client’s networking team, it was understood that their particular WAN connectivity provided a Layer 2 Ethernet hand-off to each data center such that no router was in place on either side.  This explained why larger MTU packets were traversing the WAN and being fragmented in the process.  Coordination between the WAN provider and the remote data center’s networking team was required to determine where in the network path a device was unable to handle even a standard 1500 byte packet properly.

Ultimately, there were a few options for remediation at this client in the form of either obtaining jumbo frames support across their Layer 2 WAN or placing routers on either side of the WAN.  Both of these options would remove the requirement for non-standard configurations on the actual servers while still resolving the issue of communications between the data centers.

Finding Success with Exchange 2010, by David Carlson, was published by the International Legal Technology Assosication (ILTA) as part of the March 2011 white paper series.  If you are thinking about an Exchange 2010 project, give it a read.

The article contains guidance and discussion on the following topics:

• Planning Exchange 2010 deployment for law firms
• Features for law firms: Exchange 2010 Archiving, Litigation Holds and Search
• Large Mailboxes
• Database Availability Groups
• Load Balancing Exchange 2010 CAS Arrays
• Co-existence with Cloud Computing
• Regional Consolidation of Exchange
• Virtualization

I am thrilled to have been part of numerous successful Exchange 2010 deployments for law firms.  It is hard to believe that we have already migrated tens of thousands of Exchange mailboxes and I have enjoyed sharing this experience with many ILTA members.

I hope you enjoy reading the article and welcome hearing about your experience deploying Exchange 2010.  Please leave us a comment below!

David Carlson ILTA Kraft Kennedy KKL

The basic premise of the article is this – you can achieve high availability with Exchange 2010 without the use of Database Availability Groups (DAG) by using a combination of VMware HA and Symantec’s ApplicationHA.  Application HA leverages APIs exposed in vSphere 4.1 that let third party applications interact and communicate with VMware HA to perform various functions such as restarting the guest if the entire application has failed.  ApplicationHA can also be used to restart individual guest services, so it can provide a level of application awareness beyond what VMware HA can do alone.

Joe and I agree that the blog is technically correct in that you can get high availability for Exchange 2010 without using DAGs if you use these tools.  Where we don’t agree with the author is in the usefulness of this kind of solution compared to the functionality that a DAG actually provides.

More than just HA

DAGs provide high availability of the Exchange environment but they do much more than that.  They facilitate DR scenarios by including a native replication technology that lets organizations easily replicate their email data to another site that can be brought online quickly and safely.  You can also easily move databases between Mailbox servers within the same site for routine system maintenance, Windows patches, etc.  DAGs can also detect a failure and activate another database copy quickly, most times (but not always) quicker than a VM can reboot.  Additionally, because a reboot can be required with an ApplicationHA/VMware HA solution, there is technically an outage to the end users and the end user experience associated with a reboot repopulating the Exchange database cache isn’t ideal.

DAGs provide high availability at the storage layer

Since a DAG allows up to 16 copies of any mailbox database, it can provide high availability at the storage layer in addition to just protecting the server and Exchange services.  This allows you to put your databases on completely separate storage platforms for a higher degree of availability and protection from many kinds of storage-related failures.  Since Exchange 2010 enables and encourages larger mailboxes and databases, recovery time in the event of catastrophic server failure can be significant if some kind of storage high availability or resiliency isn’t in place.

Management separation

In many firms, the engineering team responsible for Exchange may not be the same as the team responsible for the virtual environment.  In these cases, it is important to understand that leveraging a DAG would allow the Exchange team to be wholly responsible for high availability of the Exchange application.  Leveraging ApplicationHA/VMware HA to protect Exchange would require coordination between the Exchange and virtualization teams for any high availability issues/troubleshooting and that may not be appropriate or optimal in many environments.

Microsoft supported technology

Whenever you go down the road of protecting Microsoft applications with tools that are provided by third party vendors, you run the risk of having issues with Microsoft support.  This used to be more of an issue in the earlier days when host based replication tools were used to protect Exchange, SQL, etc., and is likely less of a problem these days.  I suspect that Symantec has done the right thing here and ApplicationHA does not cause issues with Microsoft support, but having never used it I can’t say for sure.

Cost

When it comes to protecting Tier-1 applications like Exchange cost is important but is not often the driving factor in making a technology decision.  Email has become one of the most important applications in the organization so firms are more willing to pay up to protect it.  That said, the cost of Symantec’s ApplicationHA seems high compared to the costs of implementing a DAG.  With a DAG, you’re just paying for an extra Windows license (which may not have a capital cost depending on your Windows licensing model) and an additional Exchange license.  That may come out cheaper than ApplicationHA, which is licensed in packs of 5 protected servers.  Again cost is usually not the most important factor, but paying less and getting more functionality out of a DAG makes it an attractive option.

Final thoughts

I strongly believe that vSphere provides a great platform for virtualizing Tier-1 applications like Exchange.  Joe and I have worked with clients that have virtualized Exchange 2010 for thousands of users on top of vSphere with great success.  Yet we both feel that the value that DAGs add to Exchange 2010 makes using them on your virtualized Exchange 2010 deployment well worth the added planning, storage requirements, and restrictions.

In my opinion there are some areas where using ApplicationHA and VMware HA guest level monitoring might be useful, such as:

• Small environments where there is no budget for additional storage purchases to accommodate multiple database copies.
• There is a requirement for using vMotion on Mailbox servers (which is not supported in combination with DAGs).  I personally cannot think of a use case where I would want to use vMotion over moving databases between nodes in a DAG, but I recognize there may be situations where this may be desirable.
• An organization already has an investment in a host based or storage based replication product for Exchange but still wants server high availability.  Again, with the native replication functionality of Exchange 2010 (and the 3rd party replication API for the DAG) there are less reasons to use features other than what the DAG provides natively.

I believe that ApplicationHA and using VMware HA for guest level monitoring is a huge step in the right direction.  There are many other use cases for this technology and I think we’ll start to see more people enabling this for nearly all workloads, particularly the native VM monitoring provided by VMware HA.  In the case of Exchange 2010, I think the benefits of DAGs outweigh the benefits that using ApplicationHA provide.

While there were a number of technical reasons for this problem, a primary reason was changes to named properties in Exchange 2010.  To avoid legacy issues with finite numbers of named properties (see my previous blog post for some more information), functionality was changed in Exchange 2010 such that named properties are now stored per mailbox instead of per database and anonymous headers are not promoted so as to avoid issues with reaching the finite limits.  Since BES leverages a number of named properties for its own functionality, the latter caused some significant performance delays when BES attempted to query non-promoted named properties.

Microsoft and RIM have been working very closely together to remediate this issue and I’m happy to announce that some significant progress has been made.  If you are experiencing this problem currently, upgrading to MAPI/CDO 1.2.3 on your BES (see here) and BES itself to 5.0.2 Maintenance Release 4 (see here) should provide significant improvement on the BES functionality side.  Additional updates from Microsoft and RIM will be coming in the near future but, in the interim, these updates should help improve performance dramatically.

Update: The new MAPI/CDO was released as an updated version of 1.2.1, not version 1.2.3 as expected.  When downloading and installing the new MAPI/CDO, make sure you are installing version 1.2.1 dated 2/25/2011 and versioned 6.5.8211.0 (not the one dated 12/9/2009 and versioned 6.5.8147).  The link above will direct you to the correct version.

Taking advantage of our Kraft Kennedy research lab, within a day I was able to connect our research iManage 8.5 system to our Exchange 2010 environment, and enabled the Archive Mailbox for my test account.     And for the first time in the history of mankind (I can only assume), tested the functionality.   Below you will find a brief summary of the environment and tests performed.

Environment

• Exchange 2010 SP1 with multiple mailbox servers behind a dedicated CAS array
• iManage WCS 8.5 SP1 Update 1
• iManage WorkSite Server 8.5 Sp1 Update 3
• Outlook 2010 with FileSite 8.5 SP2 Update 2 and EMM for FileSite 8.5 SP2 Update 2

Tests and Results

The first thing to know about the Archive Mailbox is that by default, there are no folders within (besides a Deleted Items folder).  If retention or archive rules are configured in the environment, a folder structure could be copied from the Active Mailbox to the Archive Mailbox.   For the purpose of my testing, I used a variety of folder types and drag/drop filing operations.

My first test was simply creating a folder within the Archive Mailbox, and placing an email into that folder.  I then dragged this to a Workspace folder, and the EFS back-end successfully filed the email.   1 for 1.

My next test was to create a folder in the archive, and then link that folder to a Workspace folder.  I then dragged an email from the Archive mail folder into the linked Archive Folder, and EFS successfully filed the email.  2 for 2.   I then took an email from my Active Mailbox inbox and dragged into the Linked Archive Folder.   3 for 3.  Hey, things are looking pretty good.

My last test of the day is when I started feeling a little brave.  It was time to put on the “What would an attorney do?” hat, and tried the following:   I created a Linked folder in my Active Mailbox.  I then dragged this entire folder into the Archive Mailbox.    I then tried dragging an email into this folder.  No go.  Here’s what I noticed:  although the folder icon still appeared to be a “Linked” folder, the Properties -> File to WorkSite tab had lost the association.  After re-configuring the folder to file to the proper Workspace, functionality was restored.

This behavior could also be mimicked by a backend archive rule as it archives an active mailbox folder.  Even if the link to iManage was reconfigured, the Exchange message-class should still indicate that the email has been successfully filed into WorkSite, so there shouldn’t be any duplicates.   I think the action item for Autonomy would be to see if it’s possible to change the folder icon back to a regular Outlook folder if the link to iManage is broken.

All in all, I must admit I was rather surprised at how much worked.  Part of why it works can be explained by Microsoft’s TechNet:

In Exchange 2010 SP1, when you assign Full Access permissions to a mailbox, the delegate to which you assign the permissions can also access the user’s personal archive.

So, as long as your EFS service account has the proper permissions to the user’s Active Mailbox, it will also have permissions to the Archive.    Thank you, Microsoft!   This whole experience also demonstrates the value of  a well-maintained research sandbox in your environment.

My journey technically began in early 2010, when I discussed the possibility of attending MCM training with my firm’s management.  Later in 2010, I was informed that my firm would like to send me to this training and I began the application process.  Acceptance into the MCM program as a candidate occurs only after you apply and provide a few key documents for review by the program’s team.  Specifically, you must hold an MCITP: Enterprise Messaging Administrator certification and provide a current resume or CV, an outline of a recent project describing your specific roles and responsibilities, and a technical architecture document that you’ve written for a recent project.  You may be asked to participate in a phone interview to clarify any of your materials, although I was not.  Once all required materials have been submitted, you wait to hear back from the program team regarding their decision.

Once accepted, you must choose a three week rotation date (all are on-site in Redmond at Microsoft’s headquarters) and pay the rotation training fee ($18,500 as of the writing of this article).  As you approach your rotation date, you are provided with a substantial pre-reading list that includes a large portion of Exchange TechNet content and a number of recommended lab tasks to complete.  I strongly recommend that you build a lab environment and work through as many of these recommended lab tasks as you can.  They are extremely helpful in getting you familiar with concepts you may not have worked with previously (e.g. domain secure transport, cross-forest availability sharing, etc.) and will give you a head start on some labs that you may need to work through during your rotation.

My rotation began on January 17, 2011 and it included a diverse group of 22 individuals (plus an additional classmate completing some week 1 requirements from a previous rotation) from around the world.  Microsoft Premier Field Engineering and Microsoft Consulting Services had very strong representation in the rotation with approximately half of the candidates.

Week 1

Week 1 starts slow for the first couple of hours as everyone gets situated, receives an introduction to the program, and learns about the logistics for the next three weeks.  After that, it’s full-steam ahead for the entire week.  Class begins at 8:00am sharp each day and, on average, class ran until about 10:00pm each day during week 1.  In addition, a number of lab exercises were assigned throughout the course of each day and, as a result, most candidates remained in the lab well past midnight to complete all of the labs.  While you could certainly leave earlier, I wouldn’t recommend it since you would quickly fall behind on your labs if you did so.

Week 1 covers Client Access and Transport and is, in my opinion, the most difficult and intense content that is covered during the program.  You learn the deep technical details of AutoDiscover, proxying and redirection, namespace planning, RPC Proxy, secure Exchange publishing, Exchange routing, all phases of transport categorization, transport high availability, secured transport, and much more from two excellent instructors.  There is a ton of content to learn but I can honestly say that our instructors did an amazing job of reviewing all of the key details so that I came out with a clear understanding.  Discussing RPC Proxy concepts through analogies to elephants and shadow redundancy with paper plates certainly made for a memorable learning experience.

I spent my entire weekend studying for the week 1 written exam, proctored on the second Monday.  I split my weekend between studying all classroom content and my copious notes on Saturday and then in the lab reviewing key concepts with my fellow candidates on Sunday.  That strategy worked well for me but I know others in my class spent more and less time in the lab.  My rotation was very collaborative and working together in the lab was a great learning experience.  However, if you study better on your own, you may want to tailor your own schedule accordingly.

In total, I believe we reviewed about 1,000 slides of content and spent over 110 hours in the classroom in week 1.

Week 2

Week 2 began with our week 1 written exam, bright and early at 8:00am sharp on Monday.  The format is similar to most other MCP exams but the content is, as expected, significantly more difficult.  No matter how well you feel you understand the content of week 1, do not underestimate the difficultly of this exam.  The instructors are trying to gauge if you truly understand the underlying content and, as a result, this exam is extremely difficult.  According to many, the week 1 exam is easily the hardest of the three written exams due to both the difficulty and sheer amount of associated content.  If you don’t pass on the first try, don’t be discouraged.  Brush yourself off, prepare for week 2’s training, and worry about retaking the exam later.

Week 2 covers Mailbox and Database Internals, Storage, Database Availability Groups, and High Availability.  Again, the quality of material and instructors was outstanding.  While instruction didn’t go as late as week 1 (although we had instruction until about 9:30pm on the Monday of our week 1 exam), the amount of content covered was about the same.  Paper plates made a repeat appearance when reviewing Datacenter Activation Coordination concepts and, in general, I thoroughly enjoyed the content.  In week 2, you will learn the deep technical details of ESE, Jet, retention and archiving, performance and scalability validation, Mailbox Server role sizing, storage design and sizing, storage validation, and everything you could possibly want to know about DAGs and HA.

I took a similar approach to studying on the weekend (slide and note review plus collaborative studying in the lab) but spent a little more time in the lab than for week 1.  Again, it was a great experience to study with my fellow candidates.

In total, we reviewed about 1,000 slides and spent about 60 hours in the classroom for instruction and probably another 30-40 hours studying.

Week 3

Week 3 began with our week 2 written exam, again at 8:00am sharp on Monday.  The format is the same as week 1 although, again, much more difficult than any other Microsoft exam you’ve taken before.  Once the exam is over, you can take a quick break and then dive right back into content.

Week 3 covers a number of diverse topics and is designed to tie a number of concepts together.  In week 3, you’ll cover Unified Messaging, Operations, Virtualization, RBAC, Migration and Coexistence, Load Balancing, Federation, and Multi-Forest Deployments.  A good portion of the week 3 content is consultative in nature, specifically that around migration and coexistence and multi-forest deployments.  Many of the instructors for week 3 are Solution Architects at Microsoft, who focus heavily on the consultative side of Exchange.

Due to the course’s timing, you don’t have your third weekend for studying.  The week 3 written exam was held on Friday afternoon and, while you are taking the exam, your lab environment is being rebuilt in preparation for the final qualification lab exam, which is proctored on Saturday.  As a result, you need to be very careful with budgeting your in-week studying time during week 3.  You need to prepare for the week 3 written exam but, simultaneously, you need to be preparing for the qualification lab as well.

Qualification Lab

The qualification lab is the culmination of your three weeks of training and was a truly humbling experience.  While I cannot divulge a lot of information about the exam due to NDA restrictions on MCM candidates, the basic format is that you are given a number of seemingly simple tasks to complete but that have a number of items broken that prevent those tasks from completing normally.  Your goal is to fix as many tasks as possible in the 6 hours allotted to you for the lab.  The exam is not designed to be completely finished, so don’t be overwhelmed when you start working on it.  Given that, don’t spend too much time on any single task.  If you find yourself getting stuck, switch to another task and come back later.

Many of the instructors recommended working collaboratively with fellow candidates to prepare for the qualification lab and I couldn’t agree with that more.  Each candidate has had a different set of experiences that led him or her to this course and each may have seen or troubleshot different types of issues.  Pooling your shared knowledge and helping each other not only increases your chances of passing the qualification lab but also helps strengthen your understanding of topics on which you feel confident.

Once the qualification lab is over, take the time to unwind and celebrate with your class.  My class and I arranged an excellent steak dinner at Metropolitan Grill in Seattle and had a great time celebrating the completion of our rotation.  We even saw Bill and Melinda Gates at a table right outside of our private room!

You must wait about a week to learn the results of your qualification lab but, when you finally receive that email with your results, it is truly an amazing feeling.  It is great to see all of your hard work and dedication translate into something that not many other people in the world have been able to accomplish.

Final Thoughts

Overall, this program is truly an amazing experience and one that I strongly recommend pursuing if you have interest and are able.  However, make no mistake that this is an extremely difficult course and requires a tremendous amount of personal sacrifice.  Being away from my wife for three straight weeks and, on top of that, having limited time each night to talk on the phone due to the sheer volume of work and studying in addition to the 3 hour time difference was extremely difficult.  I am very grateful to have such a supportive and encouraging wife, family, and friends to help me through the course.

To give yourself the best chance for success, complete each and every lab exercise assigned to you, including the optional ones.  While no one will be checking that you’ve completed them, anything that is covered in class is a potential topic for written exams and the qualification lab.  While it involves a lot of effort, completing all of the lab exercises will put you in the best position to succeed in the course.

Get to know your fellow candidates.  Each rotation is a diverse group of extremely intelligent and talented individuals, each with different experiences.  I am honored to have worked with such great people in my rotation and am looking forward to staying in touch for years to come.

For more information about the MCM program, please visit http://www.microsoft.com/learning/en/us/certification/master.aspx.  I’m also happy to answer any questions that I can but please understand that I may not be able to disclose some details.

Microsoft Certified Master (MCM) represents the highest technical certification that Microsoft offers and is awarded based upon successful completion of an intensive three week training course on-site at Microsoft’s headquarters along with three written exams and a final qualification lab exam.  Joe joins an elite group of  approximately 19 MCMs on Exchange 2010 worldwide to date. The Microsoft Certified Master program provides access to an exclusive community of other MCMs, members of the Exchange Server product group, as well as other valuable resources that can be leveraged at any time.

According to Microsoft, “a Microsoft Certified Master can help [an] organization assess its current messaging optimization level; map major initiatives and projects; rate the impact of identified gaps (value, cost, level of effort); and prioritize IT assets for maximum impact and return on investment.”

Joe has worked for Kraft Kennedy since 2004 and is a member of the Kraft Kennedy’s Infrastructure and Enterprise Systems practice group. He has advised numerous organizations on Exchange 2010 ranging from small firms to leading members of the AmLaw 100. Joe is also an active contributor to the Kraft Kennedy Technology Blog and has spoken at a number of ILTA events.

“When working with Joe, it is immediately clear that you have found someone with a deep understanding of the technology and also a keen awareness of the business requirements and best path to achieve them”, says David Carlson, Practice Leader at Kraft Kennedy. “This is a very exciting accomplishment and we are fortunate to have his talents as part of our team.”

You can read more about Joe and Exchange 2010 on the Kraft Kennedy Technology Blog.