Today, Microsoft has released Service Pack 2 for Exchange 2010: http://blogs.technet.com/b/exchange/archive/2011/12/05/released-exchange-server-2010-sp2.aspx
While there are a number of hotfixes and other items included, there is also some key new functionality being introduced as well:
- Cross-Site Silent Redirection for OWA – This is a tremendous improvement over current functionality where, if a firm leverages redirection between Client Access Servers in different sites (the preferred approach for optimal performance and implementation flexibility), users are prompted with a link and second authentication prompt if they login to a CAS server in a different site than where their mailbox is currently hosted. With this new functionality, Exchange can be configured silently to redirect users to the correct CAS server in this situation (without reauthentication or prompting).
- Address Book Policies – Address Book Policies provide the long-awaited native functionality support for what is typically referred to as GAL segmentation. Previously, if firms wanted selectively to exclude some users or contacts from the GAL for specific subsets of users, the firm would need to create and manage security ACLs directly within ADSI. This was unsupported in Exchange 2003 and 2010 and only narrowly supported for Exchange 2007 (with Dave Goldman’s specific whitepaper). Address Book Policies will provide an object and policy based method for providing this functionality.
- OWA Mini – This will be a lightweight, text-only version of OWA targeted for use on mobile devices or in low bandwidth/resolution scenarios.
- Hybrid Configuration Wizard – This wizard will significantly reduce the number of steps required to streamline the process for establishing rich coexistence between an on-premises Exchange 2010 environment and Office 365 (formerly BPOS).
Due to the nature of the new features included, there is an Active Directory schema update required for SP2.
Recently, Microsoft added vSphere 5.0 as a supported hypervisor for Exchange 2010 to their Server Virtualization Validation Program support policy wizard. Exchange 2010 RTM and SP1 are both listed as supported running on Windows 2008 RTM and Windows 2008 R2 on vSphere 5.0. In addition, Exchange 2007 SP1, SP2 and SP3 are listed as supported on vSphere 5.0 as well but RTM is not.
Great news!
While most of our client deployments have gone quite smoothly from the perspective of stretching Exchange 2010 Database Availability Groups across multiple sites and WAN connectivities, I recently found myself troubleshooting an inconsistent issue at one client. This environment’s topology was fairly straightforward, with two DAG members in one data center for local high availability and one DAG member in an alternate data center for remote site resiliency. Creating the DAG, adding members, and adding mailbox database copies all presented no issues during the initial deployment although we did need to resolve some issues with database copy replication across the WAN.
As we approached our anticipated IT pre-pilot for the new Exchange 2010 environment, we started to notice significant issues in DAG communications across the WAN. Specifically, we saw the following issues fairly consistently although, at some times, everything worked just fine:
- From the primary data center, viewing the mailbox database and associated copy status from the Exchange Management Console listed mount states for some databases as “Unknown” and copy status for all remote database copies as “ServiceDown.” Running Get-MailboxDatabaseCopyStatus against the DAG member(s) in the remote data center reflected the same results. Databases in an “Unknown” mount state corresponded to cases where the database was activated in one data center and status was being queried across the WAN from the other data center.
- Running “Get-DatabaseAvailabilityGroup -Status” would take an extremely long time to complete.
- Occasionally, databases would be listed in a dismounted state and, upon attempting to mount, an error message stating “Automount consensus not reached” would be returned and the mount would fail.
- Event logs on DAG members in both data centers would report sporadic occurrences of FailoverClustering events reporting that nodes in the repsective remote data center had been removed from cluster membership.
- Test-ReplicationHealth against DAG members across the WAN to the remote data center reported failures for ActiveManager (“Active Manager is in an unknown state”) and TasksRpcListener (“An error occurred while communicating with the Microsoft Exchange Replication service to test the health of the Tasks RPC Listener”).
The issue was clearly related to RPC requests traversing the WAN and having issues somewhere along the path from source to destination. As a next step, I ran the “Validate a Configuration Wizard” for the DAG’s underlying Windows Failover Cluster and, sure enough, RPC errors were reported for queries that needed to cross the WAN to talk to cluster nodes not in the same data center as the node on which the wizard was run. At this point, it was time to install Wireshark and run packet captures on either side of the WAN while executing Exchange actions or the cluster validation wizard to determine what was happening to the traffic.
Upon review of the packet captures, it was revealed that packets were being sent between DAG/cluster members that were larger than a standard 1500 byte packet and those packets were being fragmented in transit from source to destination. Disabling various large TCP offload functionality of the NIC driver in use within the DAG/cluster members (vmxnet3 Ehternet Adapter) helped to bring the packet size down to 1500 bytes but the problems still occurred. Running ping tests between the two data centers (ping -f -l <packet_size> hostname) revealed that the largest packet succeeding across the WAN was 1468 bytes. Once the MTU of the NIC was reduced to match this value (via NETSH), everything began working perfectly. The “Validate a Configuration Wizard” for the cluster completed without any unexpected warnings and all Exchange-related functionality was restored.
While disabling various functionality on the NIC and reducing the NIC’s MTU worked to solve the problem, it was certainly not ideal nor a long term solution for this environment. Ultimately, determining where the issue lied in the WAN environment was key to identify how to resolve this issue without requiring non-standard configurations on various servers in the environment. In working with the client’s networking team, it was understood that their particular WAN connectivity provided a Layer 2 Ethernet hand-off to each data center such that no router was in place on either side. This explained why larger MTU packets were traversing the WAN and being fragmented in the process. Coordination between the WAN provider and the remote data center’s networking team was required to determine where in the network path a device was unable to handle even a standard 1500 byte packet properly.
Ultimately, there were a few options for remediation at this client in the form of either obtaining jumbo frames support across their Layer 2 WAN or placing routers on either side of the WAN. Both of these options would remove the requirement for non-standard configurations on the actual servers while still resolving the issue of communications between the data centers.
Over the last few months, there have been numerous reports of significant BES message delivery delays when running Exchange 2010, sometimes in excess of 20 minutes. In addition to the primary symptom of message delivery delays to BES handhelds, environments experiencing this issue saw extremely high RPC Averaged Latency during periods of high BES utilization. Despite being generously sized for RAM on Exchange 2010 Mailbox servers already, many firms dramatically increased RAM in their Mailbox servers to compensate for this issue, with amounts totaling 36 GB for 500 active BES users in many cases. More information on some of the history and user experience with this issue can be found here.
While there were a number of technical reasons for this problem, a primary reason was changes to named properties in Exchange 2010. To avoid legacy issues with finite numbers of named properties (see my previous blog post for some more information), functionality was changed in Exchange 2010 such that named properties are now stored per mailbox instead of per database and anonymous headers are not promoted so as to avoid issues with reaching the finite limits. Since BES leverages a number of named properties for its own functionality, the latter caused some significant performance delays when BES attempted to query non-promoted named properties.
Microsoft and RIM have been working very closely together to remediate this issue and I’m happy to announce that some significant progress has been made. If you are experiencing this problem currently, upgrading to MAPI/CDO 1.2.3 on your BES (see here) and BES itself to 5.0.2 Maintenance Release 4 (see here) should provide significant improvement on the BES functionality side. Additional updates from Microsoft and RIM will be coming in the near future but, in the interim, these updates should help improve performance dramatically.
Update: The new MAPI/CDO was released as an updated version of 1.2.1, not version 1.2.3 as expected. When downloading and installing the new MAPI/CDO, make sure you are installing version 1.2.1 dated 2/25/2011 and versioned 6.5.8211.0 (not the one dated 12/9/2009 and versioned 6.5.8147). The link above will direct you to the correct version.
I recently had the opportunity to attend Rotation 8 of the Microsoft Certified Master | Exchange 2010 program and am happy to report that I passed all three written exams and the final qualification lab exam on the first attempt. I’m proud to be joining an elite community of only 19 MCMs worldwide prior to my rotation and those of my Rotation 8 colleagues that have passed or will after exam retakes. I had the privilege of learning with some extremely talented individuals in my rotation and am looking forward to working with other MCMs in the near future. I wanted to share some of my experiences in the hopes that it helps encourage others to pursue this elite certification.
Continue reading…
Research In Motion has released new guidance for the BlackBerry Enterprise Server (BES) service account after upgrading to Exchange 2010 Service Pack 1. While BES does not yet formally support SP1, it can work with a few changes to the custom Throttling Policy created for the BES service account. Please note that, while BES can work with Exchange 2010 SP1, you must strongly consider RIM’s current support statement before completing an upgrade to SP1. It is strongly encouraged to wait for formal support from RIM before upgrading a production Exchange 2010 environment to SP1 if BES is a critical application in your environment.
Continue reading…
I’ve worked with a few clients that, for various reasons, have modified the FQDN of the SMTP Virtual Server. By default, an SMTP Virtual Server will respond to an EHLO with the FQDN of the server itself (e.g. NYMAIL01.client.local) but some clients have adjusted this to something entirely different (e.g. SMTP.client.com). Most of the time, clients have changed this to mask the name of the underlying server responding to SMTP.
While modifying the FQDN of the SMTP Virtual Server certainly does mask this, the actual internal IP address of the server is still listed in the e-mail message headers and, as such, the identity of the server isn’t actually masked. Furthermore, if a hacker were to compromise a perimeter firewall, not knowing the exact name of the server providing SMTP services will not deter much.
Continue reading…
Client throttling is a feature of Exchange 2010 that restricts simultaneous connections and processor utilization on a per user basis so that a single user or rogue process cannot exhaust precious resources on the Exchange server. However, if not configured properly and adjusted to meet an individual environment’s needs, client throttling can lead to end user frustration and the inability to complete work functions. Microsoft’s TechNet article here describes client throttling and details for each configurable parameter. The most common issues associated with client throttling that I’ve seen in client environments are related to delegate mailbox access and third party integrated applications, which I discuss below.
Continue reading…
Once you have moved all of your mailboxes to Exchange 2010 and properly decommission your Exchange 2003 (move public folder hierarchies, transition mail flow, move OAB generation, etc.), you may see MSExchange Store Driver event ID 1020 errors like these:
The store driver couldnt deliver the public folder replication message “Hierarchy (PublicFolder@client.com)” because the following error occurred: The Active Directory user wasn’t found.
These errors likely coincide with some public folder replication issues but, more importantly, can also result in NDRs when messages are sent to mail-enabled public folders! This issue occurs because, even if you properly decommission Exchange 2003, the Servers containers within your legacy Exchange 2003 Administrative Groups still exist within Active Directory, albeit empty. Exchange assumes that, if a Servers container exists (even if it is empty), a System Attendant object will also exist somewhere inside of it but, if all of your Exchange 2003 servers have been decommissioned, those System Attendant objects actually do not exist.
This issue has been recognized as a bug within Exchange (see the MS Exchange Team Blog article here) and a fix a scheduled for an upcoming Update Rollup release (perhaps Update Rollup 5). In the meantime, you can safely delete these empty Servers containers via ADSI Edit but make sure that these containers completely empty before doing so.
For more in my series on Exchange 2010 Notes from the Field, please click here.
One of my earlier Exchange 2010 deployments was at a client that had modified the default inheritance settings of Active Directory such that default security permissions did not apply to some Organizational Units (OUs). This prevented ActiveSync from creating necessary objects and setting necessary attributes to provision iPhones for these users against their Exchange 2010 mailboxes. Similar issues occur if you attempt to configure an ActiveSync device for a mailbox associated with a user that is a member of certain privileged groups within Active Directory (e.g. Domain Admins, Enterprise Admins, etc.).
To resolve this issue for the specific case at my client, we simply needed to enable inheritance on the OUs or users where it had previously been disabled.
Resolving this issue for members of privileged groups is a bit more complicated. Basically, the lack of inheritance is by design for users that are members of privileged AD groups. Every hour, a background process runs on domain controllers to apply the permissions assigned to the AdminSDHolder template object to all members of privileged groups. You can review the permissions that will be applied by launching Active Directory Users and Computers, enabling Advanced Features within the View menu, and then reviewing the security permissions of the AdminSDHolder object within the System OU.
The true solution is to provide administrators with separate administrative-only accounts (e.g. JohnAdmin.admin) that are members of the required AD groups and have these administrators use normal, non-privileged accounts (e.g. JohnAdmin) for e-mail functionality. In some environments, this may not be possible and, as a result, you have two workarounds. First, you could modify the permissions on the AdminSDHolder template object to include the required Exchange permissions. I don’t recommend this since you would be modifying a fairly important and engrained aspect of Active Directory for what should be a few isolated users. Instead, you could temporarily enable inheritance on your administrative users and, as long as you configure these users’ ActiveSync devices before the next application of AdminSDHolder permissions, it will work just fine. Once an ActiveSync device is provisioned for the user, these special Exchange permissions are no longer required.
For more information on AdminSDHolder, the associated default permissions, and instructions for modifying these permissions, please refer to http://policelli.com/blog/?p=136.
For more in my series on Exchange 2010 Notes from the Field, please click here.
0
Back to top