Welcome to Kraft Kennedy

Join Kraft Kennedy’s Joe Hoegler for an ILTA Webinar entitled “Exchange 2010 Ecosystem”. The webinar will focus on the native features of Microsoft Exchange 2010, including high availability, disaster recovery, antivirus and antispam functionality, unified messaging and more. You’ll also hear guidelines and tips on integrating Exchange with SharePoint 2010, Office 2010 and Lync Server 2010.

When: Friday, February 24, 2012
Time: 12:00-1:00 Eastern
Register: Click Here to Register

Presenter BIO:

Joe Hoegler is the Practice Leader of Kraft Kennedy’s Infrastructure and Enterprise Systems Practice Group. He provides technical leadership and strategic guidance on client engagements involving a broad range of law firm technologies and is responsible for directing technology strategy and providing technical management at the firm. As of December 2011, he has led or advised over 35 law firm clients totaling over 35,000 users on projects related to Exchange 2010, ranging in size from 30 to 6,000 users. Joe is a Microsoft Certified Master on Exchange 2010, one of only approximately 50 people worldwide who hold this certification.

One of the shortcomings of Apple’s iOS devices (as of iOS 5.0.1) is the inability to recognize message priority flags.  So if a user with an iPhone receives a message that has been sent with high priority, there is no native feature in the iOS operating system that will alert the recipient to the fact that they have such a message.  However, for iOS users who utilize Microsoft Exchange for their corporate email, there is a way to bring special attention to messages sent with high priority via SMS.

The first step is to create a contact that will serve as the recipient for the SMS message.  You will need to consult the settings for your particular carrier to find the format for this address.  For AT&T users, the format will be [10-digit phone number]@txt.att.net.  Once the contact is created, create an email rule similar to the one below:

Continue reading…

While most of our client deployments have gone quite smoothly from the perspective of stretching Exchange 2010 Database Availability Groups across multiple sites and WAN connectivities, I recently found myself troubleshooting an inconsistent issue at one client.  This environment’s topology was fairly straightforward, with two DAG members in one data center for local high availability and one DAG member in an alternate data center for remote site resiliency.  Creating the DAG, adding members, and adding mailbox database copies all presented no issues during the initial deployment although we did need to resolve some issues with database copy replication across the WAN.

As we approached our anticipated IT pre-pilot for the new Exchange 2010 environment, we started to notice significant issues in DAG communications across the WAN.  Specifically, we saw the following issues fairly consistently although, at some times, everything worked just fine:

• From the primary data center, viewing the mailbox database and associated copy status from the Exchange Management Console listed mount states for some databases as “Unknown” and copy status for all remote database copies as “ServiceDown.”  Running Get-MailboxDatabaseCopyStatus against the DAG member(s) in the remote data center reflected the same results.  Databases in an “Unknown” mount state corresponded to cases where the database was activated in one data center and status was being queried across the WAN from the other data center.
• Running “Get-DatabaseAvailabilityGroup -Status” would take an extremely long time to complete.
• Occasionally, databases would be listed in a dismounted state and, upon attempting to mount, an error message stating “Automount consensus not reached” would be returned and the mount would fail.
• Event logs on DAG members in both data centers would report sporadic occurrences of FailoverClustering events reporting that nodes in the repsective remote data center had been removed from cluster membership.
• Test-ReplicationHealth against DAG members across the WAN to the remote data center reported failures for ActiveManager (“Active Manager is in an unknown state”) and TasksRpcListener (“An error occurred while communicating with the Microsoft Exchange Replication service to test the health of the Tasks RPC Listener”).

The issue was clearly related to RPC requests traversing the WAN and having issues somewhere along the path from source to destination.  As a next step, I ran the “Validate a Configuration Wizard” for the DAG’s underlying Windows Failover Cluster and, sure enough, RPC errors were reported for queries that needed to cross the WAN to talk to cluster nodes not in the same data center as the node on which the wizard was run.  At this point, it was time to install Wireshark and run packet captures on either side of the WAN while executing Exchange actions or the cluster validation wizard to determine what was happening to the traffic.

Upon review of the packet captures, it was revealed that packets were being sent between DAG/cluster members that were larger than a standard 1500 byte packet and those packets were being fragmented in transit from source to destination.  Disabling various large TCP offload functionality of the NIC driver in use within the DAG/cluster members (vmxnet3 Ehternet Adapter) helped to bring the packet size down to 1500 bytes but the problems still occurred.  Running ping tests between the two data centers (ping -f -l <packet_size> hostname) revealed that the largest packet succeeding across the WAN was 1468 bytes.  Once the MTU of the NIC was reduced to match this value (via NETSH), everything began working perfectly.  The “Validate a Configuration Wizard” for the cluster completed without any unexpected warnings and all Exchange-related functionality was restored.

While disabling various functionality on the NIC and reducing the NIC’s MTU worked to solve the problem, it was certainly not ideal nor a long term solution for this environment.  Ultimately, determining where the issue lied in the WAN environment was key to identify how to resolve this issue without requiring non-standard configurations on various servers in the environment.  In working with the client’s networking team, it was understood that their particular WAN connectivity provided a Layer 2 Ethernet hand-off to each data center such that no router was in place on either side.  This explained why larger MTU packets were traversing the WAN and being fragmented in the process.  Coordination between the WAN provider and the remote data center’s networking team was required to determine where in the network path a device was unable to handle even a standard 1500 byte packet properly.

Ultimately, there were a few options for remediation at this client in the form of either obtaining jumbo frames support across their Layer 2 WAN or placing routers on either side of the WAN.  Both of these options would remove the requirement for non-standard configurations on the actual servers while still resolving the issue of communications between the data centers.

Back in February I read a blog post from Andre Leibovici (VDI and Microsoft Outlook, analysing the variables), a very well known and respected expert in the virtualization and VDI community.  His article discussed the challenges in dealing with Outlook in VDI environments, including how to address OST/PST files and how searching is affected.  Although I’m a little late to the party here I thought I’d add my thoughts on this and make sure folks aren’t seeing this as a barrier to adopting VDI.

I recommend reading Andre’s post to get more information on this topic.  The short version is this – Exchange/Outlook best practices do not necessarily work in a VDI environment.

When using Outlook in an Exchange environment, it is recommended to use Cached Exchange Mode.  In this mode a copy of the user’s mailbox is downloaded into an OST file and stored offline on the user’s desktop.  This mode offers better performance for the end user and reduces utilization on the Exchange environment as well.  In addition, using Cached Exchange Mode allows users to use Outlook Instant Search for fast searching of items in their mailbox.  Instant Search works by indexing the contents of the OST file so all searches occur locally and not on the Exchange server, further improving performance and reducing utilization on Exchange.

VDI environments that we see at our clients are typically configured as non-persistent or floating pools of desktops.  That is, each user connects to a pool of identical desktops and grabs whatever desktop is available.  When the user logs off, any changes written to the VDI desktop are discarded and the desktop returns to a pristine state.  There are mechanisms and tools in place to make sure user data is retained at logoff.

So if user data is retained at logoff, why can’t we use Cached Exchange Mode in non-persistent VDI environments?
Continue reading…

Kraft Kennedy is pleased to announce achievement in 8 Microsoft Gold and Silver Competencies (and counting!) for 2011.

The requirements to participate in the Microsoft Partner Program have recently evolved to help differentiate technical and business capabilities among participants; Kraft Kennedy has risen to the challenge by quickly exceeding the goals set forth by the program.

Each competency requires specific individuals with deep technical skills, Microsoft verified customer references, and challenging certification exams to be completed.  This commitment demonstrates our breadth, deep specialization, and proven expertise across a range of Microsoft technologies.

About Kraft Kennedy

Kraft Kennedy provides business and technology-related consulting services to the legal community. By combining outstanding technical skills with an intimate knowledge of our clients’ business and information needs we tailor solutions that enhance attorney productivity, effectiveness, and client value.

We focus on the business needs of the client and ensure that technology is used to enhance, not inhibit their business. KK’s talented staff of strategic consultants, project managers, and network consultants have years of experience with hundreds of projects for firms from small to large. Our services portfolio includes advanced infrastructure projects, business continuity and data center consolidation, desktop deployment, network design and implementation, storage design and replication, and messaging systems migration among others. Our Microsoft specialties include: Desktop, Server Platform, Unified Communications, Portals and Collaboration, Search, Systems Management, Virtualization, and Small Business Specialist Community.

I’ve worked with a few clients that, for various reasons, have modified the FQDN of the SMTP Virtual Server.  By default, an SMTP Virtual Server will respond to an EHLO with the FQDN of the server itself (e.g. NYMAIL01.client.local) but some clients have adjusted this to something entirely different (e.g. SMTP.client.com).  Most of the time, clients have changed this to mask the name of the underlying server responding to SMTP.

While modifying the FQDN of the SMTP Virtual Server certainly does mask this, the actual internal IP address of the server is still listed in the e-mail message headers and, as such, the identity of the server isn’t actually masked.  Furthermore, if a hacker were to compromise a perimeter firewall, not knowing the exact name of the server providing SMTP services will not deter much.
Continue reading…

Client throttling is a feature of Exchange 2010 that restricts simultaneous connections and processor utilization on a per user basis so that a single user or rogue process cannot exhaust precious resources on the Exchange server.  However, if not configured properly and adjusted to meet an individual environment’s needs, client throttling can lead to end user frustration and the inability to complete work functions.  Microsoft’s TechNet article here describes client throttling and details for each configurable parameter.  The most common issues associated with client throttling that I’ve seen in client environments are related to delegate mailbox access and third party integrated applications, which I discuss below.
Continue reading…

Once you have moved all of your mailboxes to Exchange 2010 and properly decommission your Exchange 2003 (move public folder hierarchies, transition mail flow, move OAB generation, etc.), you may see MSExchange Store Driver event ID 1020 errors like these:

The store driver couldnt deliver the public folder replication message “Hierarchy (PublicFolder@client.com)” because the following error occurred: The Active Directory user wasn’t found.

These errors likely coincide with some public folder replication issues but, more importantly, can also result in NDRs when messages are sent to mail-enabled public folders!  This issue occurs because, even if you properly decommission Exchange 2003, the Servers containers within your legacy Exchange 2003 Administrative Groups still exist within Active Directory, albeit empty.  Exchange assumes that, if a Servers container exists (even if it is empty), a System Attendant object will also exist somewhere inside of it but, if all of your Exchange 2003 servers have been decommissioned, those System Attendant objects actually do not exist.

This issue has been recognized as a bug within Exchange (see the MS Exchange Team Blog article here) and a fix a scheduled for an upcoming Update Rollup release (perhaps Update Rollup 5).  In the meantime, you can safely delete these empty Servers containers via ADSI Edit but make sure that these containers completely empty before doing so.

For more in my series on Exchange 2010 Notes from the Field, please click here.

One of my earlier Exchange 2010 deployments was at a client that had modified the default inheritance settings of Active Directory such that default security permissions did not apply to some Organizational Units (OUs).  This prevented ActiveSync from creating necessary objects and setting necessary attributes to provision iPhones for these users against their Exchange 2010 mailboxes.  Similar issues occur if you attempt to configure an ActiveSync device for a mailbox associated with a user that is a member of certain privileged groups within Active Directory (e.g. Domain Admins, Enterprise Admins, etc.).

To resolve this issue for the specific case at my client, we simply needed to enable inheritance on the OUs or users where it had previously been disabled.

Resolving this issue for members of privileged groups is a bit more complicated.  Basically, the lack of inheritance is by design for users that are members of privileged AD groups.  Every hour, a background process runs on domain controllers to apply the permissions assigned to the AdminSDHolder template object to all members of privileged groups.  You can review the permissions that will be applied by launching Active Directory Users and Computers, enabling Advanced Features within the View menu, and then reviewing the security permissions of the AdminSDHolder object within the System OU.

The true solution is to provide administrators with separate administrative-only accounts (e.g. JohnAdmin.admin) that are members of the required AD groups and have these administrators use normal, non-privileged accounts (e.g. JohnAdmin) for e-mail functionality.  In some environments, this may not be possible and, as a result, you have two workarounds.  First, you could modify the permissions on the AdminSDHolder template object to include the required Exchange permissions.  I don’t recommend this since you would be modifying a fairly important and engrained aspect of Active Directory for what should be a few isolated users.  Instead, you could temporarily enable inheritance on your administrative users and, as long as you configure these users’ ActiveSync devices before the next application of AdminSDHolder permissions, it will work just fine.  Once an ActiveSync device is provisioned for the user, these special Exchange permissions are no longer required.

For more information on AdminSDHolder, the associated default permissions, and instructions for modifying these permissions, please refer to http://policelli.com/blog/?p=136.

For more in my series on Exchange 2010 Notes from the Field, please click here.

In some cases, you may encounter an issue where you move a mailbox from Exchange 2003 to Exchange 2010 and, while the move request completed, you receive a warning stating “failed to cleanup the source mailbox after the move.”  All of the content is successfully moved, appropriate attributes are updated to point to Exchange 2010, and all new messages are delivered to the Exchange 2010 mailbox.  However, you will see disconnected mailboxes for the affected users in Exchange 2003 System Manager and you cannot purge the disconnected mailboxes.

The issue can be caused by search folder problems in the Exchange 2003 mailbox and similar issues were fixed in Exchange 2007 via Service Pack 2.  Microsoft describes ways to purge the disconnected Exchange 2003 mailbox at http://support.microsoft.com/default.aspx?scid=kb;EN-US;930363 but the workarounds involve either a number of manual steps or reducing your mailbox retention settings to 0 days and allowing normal Exchange online maintenance to purge the mailboxes.  The latter would also purge any legitimately deleted mailboxes that you may want to retain for some period of time so the safest workaround may be to just wait for your normal online maintenance procedures to purge the mailboxes for you (30 days by default).

For more in my series on Exchange 2010 Notes from the Field, please click here.