Quite a while back I saw that Eric Sloof had figured out how to add his Twitter feed directly into the VI Client.  I thought it was clever but didn’t really give it much more thought than that.

Today I decided to take that concept and extend it to systems that you might manage alongside your VI3/vSphere environment.  Storage management seemed like the obvious first choice.

I created an XML file called EqualLogic.xml in C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Plugins\SAN Management.  The contents of the file are as follows (you would replace the <url> section with the IP or DNS name of your SAN):

<scriptConfiguration version=”1.0.0″>
<key>EqualLogic</key>
<description>EqualLogic SAN Management</description>
<view parent=”Inventory.HostSystem”>
<title locale=”en”>EqualLogic</title>
<url>http://10.1.97.30</url>
</view>
</scriptConfiguration>

This gave me a nice way to manage my SAN from the same interface that I use to manage my vSphere environment.  It is simply opening a browser window within the vSphere Client and letting me manage the SAN.

The code above will make the EqualLogic tab visible only when clicking on an ESX/ESXi host.  If you wanted to extend that to other objects, you can simply adjust the “<view parent=” section.  For example, to also make this available at the cluster level you would include the following:

<view parent=”Inventory.Cluster”>
<title locale=”en”>EqualLogic</title>
<url>http://10.1.97.30/</url>
</view>

Similarly you could add Inventory.Global, Inventory.VirtualMachine, Inventory.Datacenter, etc.

This is a really simple way to make it easy to manage any web interface (not just EqualLogic) from within the vSphere/VI Client.  It’s not a new trick and has been out there for a while but I had never used it for this until today.

I had the privilege of attending Microsoft TechEd North America 2010 last week in New Orleans, LA and wanted to share a few confirmed new features of Exchange 2010 SP1 (due out later this year but no firm ETA).  The overall theme of this year’s TechEd was centered around, not surprisingly, cloud computing.  Specifically, Microsoft emphasized their mature Exchange Online offering but also stressed the growth of Windows Azure (Microsoft’s hosted services platform) and hosted SQL services.  The demonstrations of what Windows Azure and hosted SQL could do were extremely interesting and solidified Microsoft’s vision of being “all in” (according to keynote speaker Bob Muglia) with cloud computing.

Regarding Exchange 2010 SP1, a few of the Exchange sessions I attended had specific focus on SP1 and most of the others mentioned how different aspects of Exchange would change with SP1.  I discuss a number of my favorite changes/additions associated with SP1 below but please note that there are many more new features.  Please refer to the MS Exchange Team blog post on SP1 for more information.

Continuous Replication Block Mode

Continuous Replication Block Mode (CRBM) is, in my opinion, the most interesting and compelling new feature of Exchange 2010 SP1.  It is so important that it required a rename of the legacy continuous replication of Exchange 2007/2010 RTM to Continuous Replication File Mode (CRFM).  Basically, in CRFM or legacy continuous replication, the unit of replication in Exchange is a single log file.  Thus, after each 1 MB log file was closed on the active server (Exchange 2007) or database copy (Exchange 2010), the passive server or database copies would pull that log file and replay into the passive copy of the database.  This resulted in a worst case recovery point objective (RPO) of 1 MB (the active log stream) when replication was healthy and up-to-date.

CRBM is a dynamic continuous replication mode that is automatically turned on or off by Exchange 2010 when it detects that replication is completely up-to-date.  When turned on, CRBM allows Exchange to ship transactions committed to the active log stream to passive database copies.  Thus, the passive copies maintain their own log stream and can significantly reduce the RPO of Exchange in the event of a failure of the active copy.  However, while CRBM could bring your RPO down to individual transactions, I wouldn’t recommend advertising an RPO of less than 1 MB to a business owner since you cannot control CRBM and it may be turned off at any time.  In my opinion, it is better to advertise 1 MB and indicate that, under healthy replication circumstances, the realized RPO will likely be much better.

CRBM is not synchronous, so Exchange will not wait for an acknowledgement from the passive database copy that the log stream write succeeded.  Additionally, since a CRBM passive database copy now maintains its own log stream, it will automatically convert a partial log stream (log fragment) into a full log file for replay in the event of a failure of the active database copy.

Other High Availability and Site Resiliency Improvements

In addition to CRBM, SP1 will bring a number of other high availability and site resiliency improvements.  First, Outlook cross-site connection behavior will be more flexible by providing the option to have, in the event of a cross-site database failover, either a direct CAS connection from the CAS Array in the primary data center to the hosting Mailbox server in the secondary data center (default in RTM) or to disable this functionality entirely.  Second, Datacenter Activation Coordination (DAC) mode now becomes available for DAGs of all types, not just those with three or more members that are stretched across two or more sites.  Finally, improvements in gracefully shutting down log writes when a database activation is initiated removes the need for Exchange to perform a recovery process when a passive copy is activated.  This can reduce a typical database activation process from 30 seconds to 15 seconds.

Scott Schnoll at Microsoft has written a great article about these and other high availability and site resiliency changes in SP1 here.

Archiving and Discovery Improvements

As alluded to by Microsoft around when Exchange 2010 RTM was announced, SP1 will allow additional deployment flexibility with the Personal Archive.  First, formal support for Outlook 2007 will exist, although it will be limited to basic access to the Personal Archive.  Manipulation of retention policies and other aspects of the archive will be limited to Outlook 2010 and OWA 2010.  Second, SP1 will now allow storage of the Personal Archive in a different mailbox database than the primary mailbox.  Supported combinations of storage are primary and archive in the same mailbox database (on-premises or hosted), primary and archive in different mailbox databases (both on-premises), or primary on-premises and archive hosted.  While many law firms may find it difficult to adopt Exchange Online in the near term due to back-end application integration requirements with Exchange, the ability to host some users completely in the cloud and perhaps the Personal Archive for other users in the cloud as well is very intriguing.

Discovery becomes more robust by offering search preview to estimate anticipated discovery searches before executing a query, optional de-duplication of results in a completed discovery, and annotations for discovered content.

Outlook Web App Performance and Personalization Improvements

OWA performance has been improved dramatically in SP1 through a number of enhancements.  OWA will now pre-fetch content to make content presentation and reading faster.  Additionally, delete, mark as read, and categorization actions will now be asynchronous operations so that their results look instantaneous to the end user.  Long running operations, like large file transfers, will no longer be blocking operations that cause OWA hanging from an end user perspective.

OWA themes will make a comeback in SP1, with some prebuilt themes included and the ability to design your own (perhaps to match a corporate color scheme branding, etc.).  Furthermore, the administrator can granularly control if/how themes can be used.

Rich Coexistence with Exchange Online

Once Exchange Online is upgraded to Exchange 2010 SP1 later this year, coexistence between an on-premises solution and Exchange Online will become much more robust.  Due to the enhanced federated sharing features of Exchange 2010 (calendaring, etc.), traditional coexistence issues associated with free/busy lookups, internal mail classification, etc. are all resolved to provide much more seamless coexistence with Exchange Online.  With the ability to easily host and manage a subset of users in the cloud, this rich coexistence model makes this approach much more appealing.

As you can see, Exchange 2010 SP1 will provide many important new features to improve resiliency, flexibility, and performance of Exchange in a number of areas.  The SP1 Beta is available now, so check it out!

For quite a while there has been confusion over how VMware’s Transparent Page Sharing (TPS) feature works with vSphere 4 running on Nehalem (or other modern) processors. Many people were noticing that it appeared that TPS was not actually working anymore and looked for ways to fix the problem.

In my recent post on the effects of ASLR in vSphere the comments turned into a discussion about TPS on modern processors. And there are countless posts about this issue on the VMTN forums where folks are looking for a fix. In reality nothing is broken and there is no need to fix the issue.

VMware has published a KB article that gives more information on TPS with Nehalem processors and why it appears TPS isn’t working (this affects modern AMD processors also). The short version is that TPS uses small pages (4K), and Nehalem processors utilize large pages (2MB). The ESX/ESXi host keeps track of what pages could be shared, and once memory is over-comitted it breaks the large pages into small pages and begins sharing memory.

Many people think this is a bug in ESX that needs to be fixed. This likely started because when vSphere 4 was released there was a bug around memory usage on ESX hosts with Nehalem processors. In reality the bug was that vCenter was triggering high memory usage alarms for virtual machines running in this configuration. Nothing was actually wrong but because the host was using all of the assigned memory for the VM, vCenter was incorrectly triggering the alarm. That behavior has since been fixed with a patch and is no longer an issue.

So what does this actually look like? When a VM is powered up on an ESX host with Nehalem processors, the amount of host memory in use will not drop down as the VM uses less memory or becomes idle. Those of us that have been using ESX for a long time likely found this scenario disturbing.

From vSphere Client (red highlighted section shows guest taking all of the 2GB assigned memory, yet memory usage in the guest is very low):

From esxtop (red highlighted section shows almost no memory being shared with page sharing):

The above screenshots show a host that is under-committed on memory and so no page sharing is occurring.  If the host gets over-commited page sharing kicks in automatically by breaking up large pages into small pages.  You can force the use of small pages on all guests all the time by changing the value of the advanced option Mem.AllocGuestLargePage to 0.  I don’t really see any reason to do this – remember that TPS isn’t broken and what you see in the above screenshots is normal and expected.

Once host memory is over-committed (or if you use the advanced option), memory sharing kicks in and things look like they normally do when page sharing is taking place.

From the vSphere Client (red highlighted section shows guest taking very little of the assigned 2GB memory as page sharing has kicked in):

From esxtop (red highlighted sections show a large amount of shared memory and the host is over-commited on memory by 48%):

A quick note on the esxtop screenshot above – it was taken from a VDI environment where all workloads are identical so that explains the high amount of shared pages.  It was also overcommitted more than normal as it was taken during host maintenance.

I hope this clears up some of the confusion around TPS on modern Intel/AMD processors.  In short, don’t get hung up on the fact that TPS isn’t kicking in like it did with older processors.  Nothing is broken, TPS is working as expected, and it will kick in when you actually need it.

Wyse unveiled the “Xenith” thin client device last week at Synergy.  And unlike Wyse’s other thin client devices for Citrix that run Windows XPe or Windows CE, the “zero” client runs an ultra thin firmware (<5 Mb).  This thin firmware means the device boots up instantly and has minimal management.   A demo at Synergy last week showed the thing boot up in less than 5 seconds.  What else separates the Xenith from traditional thin client devices?   Full HDX support including HDX MediaStream (including Flash), HDX Plug-n-Play (USB redirection) and HDX RealTime (bi-directional audio). The expectation being that as Citrix upgrades and improves HDX features in the future, the Xenith’s firmware will be able to be upgraded to provide this support.  Firmware and asset management can be done through Wyse Device Manager and availability is expected in June with a price point at around $330.

The Xenith isn’t out yet, but seems very promising with HDX support, thin firmware, minimal management and an attractive price point.  If a firm is considering a VDI environment with XenDesktop in the next 6 months, the Wyse Xenith is definitely worth a look.

In Q4 last year, Citrix made its NetScaler physical appliances available as a virtual appliance.  Labeled as the “VPX”, the full featured virtual iteration of the appliance dropped its price point and made it more accessible to SMB customers.  Citrix has now made the Access Gateway (CAG) and Branch Repeater physical appliances also available as VPXs.  At this point, Citrix has made three of their ‘core’ Networking products available as VPX appliances, which are recapped below.

Citrix NetScaler VPX

• Released last year in Q4.  Initially, this virtual appliance was only released for XenServer, but now has full support for ESX.  Expected to support Hyper-V in the late summer.
• Licensed by throughput and available in 10 Mbit, 200 Mbit and 1000 Mbit flavors.
• Free “Express” edition with 1 Mbit throughput limitation available for lab and PoC environments.
• Standard NetScaler “Advanced”, “Enterprise” and “Platinum” flavors available.

Citrix Access Gateway VPX

• Released earlier this month.  Currently only supports XenServer. Support for ESX and Hyper-V expected in the next 6-12 months.
• Provides same feature as Model 2010 Access Gateway physical appliance.
• Free “Express” edition exists that provides access to 5 concurrent users that are valid for 12 months.

Citrix Branch Repeater VPX

• Released in Q1 of 2010.  Currently only supports XenServer. Support for ESX and Hyper-V expected in the next 6-12 months.
• VPX Appliance does not support the following features available in the physical appliance:
  • Group Mode
  • Ethernet bypass card
• Still requires Citrix Repeater appliance, which is not available in VPX format.

This week at Citrix Summit/Synergy, Citrix finally revealed details behind their much anticipated client (bare metal) hypervisor.  To recap, for the folks who are not following, this will finally bring “offline VDI” to XenDesktop.  It will also match (and potentially beat) VMware’s current offline VM checkin/check out functionality currently available in View.

XenClient 1.0 will be released later this week for download on MyCitrix and is being demoed and talked about at the conference.  After playing with it at one of the demo stations and talking with some Citrix Engineers, here are some details:

• Unlike VMware’s View, XenClient is a Type 1 hypervisor.  This means it lives above the client side OS (Windows).  Once installed, the user has the option to boot into whatever VMs are available on the laptop.
• At the moment, XenClient will only support a small subset of hardware types.  This includes Dell’s Latitude E series, Dell Optiplex 780, and  HP EliteBook laptops.  Full HCL to be published later in the week.
• Citrix Synchronizer is the server appliance that chats with the XenClient (over HTTP/SSL) to continuously sync and update the local running VM back to the Data Center.
• Synchronizer will be available as a virtual appliance running on XenServer.  According to Citrix, there is no planned version for ESX.  – I am sure this will change though.
• XenClient can be installed as a standalone or in conjunction with “Synchronizer”.
• XenClient supports paravirtualization to allow VMs direct access to hardware (using it’s native driver).  For example, a VM under XenClient can tap directly into a GPU for accelerated video playback and graphic intensive applications.  – This demoed very well with the engineer playing back an HD video file without skipping within a VM.
• At the moment, only a small subset of USB devices are supported through XenClient.

I am at Synergy all week, so I expect to learn more details about XenClient over the course of the week.  If you guys have questions, feel free to post them in the comments and I’ll try to get those questions answered while at the conference.

At a few recent client implementations, we have seen noticeable delays synchronizing various changes in mailboxes to Exchange 2010 when running Outlook 2003 in Online Mode.  As it turns out, this is a known issue and Microsoft has documented it at http://support.microsoft.com/kb/2009942.  The following are symptoms of the issue:

• Outgoing messages stay in the Outbox for up to 1 minute
• New messages do not arrive in the mailbox for up to 1 minute
• Items that are deleted or moved between folders may take up to 1 minute for the change to be reflected

This issue does not happen with Outlook 2003 in Cached Mode or in Outlook 2007/2010 in Online or Cached Mode.  The issue arises because Outlook 2003 requests UDP notifications from Exchange to determine when to retrieve messages.  When a new message arrives in an Exchange mailbox, Exchange 2007 and earlier would send a UDP notification to Outlook, which would then trigger Outlook to retrieve the message and display to the user.  In the absence of receiving any UDP notifications from Exchange, Outlook reverts to a scheduled polling of Exchange that occurs every 60 seconds, by default.

Exchange 2010 no longer issues UDP notifications to Outlook and, as a result, Outlook is then wholly dependent on its own polling frequency to retrieve new messages.  Outlook 2003 in Cached Mode and Outlook 2007/2010 are not affected by this issue because the former uses a different synchronization algorithm to send or retrieve content and Outlook 2007/2010 use an asynchronous notification process that does not rely on UDP notifications.

There are a few solutions and a workaround for the problem, as I note below.

• Implement Cached Mode for Outlook 2003 – This solution resolves the issue, as described above, but still leaves a legacy Outlook client in use.
• Upgrade to Outlook 2007/2010- We recommend that firms strongly consider upgrading the Outlook client to 2007/2010 when deploying Exchange 2010 due to all of the new features that require a newer Outlook client (OAB web-based distribution, Autodiscover, Personal Archive, etc.).  We also recommend Outlook in Cached Mode as a best practice, provided workstation hardware is sufficient for mailbox sizes.
• Reduce Polling Frequency of Outlook to Exchange – This involves the steps as outlined below to add a registry value on the Exchange 2010 CAS role to reduce the maximum polling frequency of Outlook.  While the registry value supports anywhere from 5 seconds to 2 minutes, Outlook 2003 cannot poll any more frequently than every 10 seconds.  As a result, this represents a best case scenario for Outlook 2003 in Online Mode and something where users will still likely notice delays.
  • Install Exchange 2010 Update Rollup 1.
  • On all Exchange 2010 CAS servers, navigate to  HKLM\System\CurrentControlSet\Services\MSExchangeRPC\ParametersSystem and create a REG_DWORD entitled “Maximum Polling Frequency” with a value from 5000 to 120000 (decimal value).
    • Note that, as mentioned above, configuring this to anything less than 10000 will have no additional effect on Outlook 2003.
  • Restart the Microsoft Exchange RPC Client Access service on your CAS servers.  You shouldn’t need to restart your Outlook clients but you may want to as well.

When VMware released vSphere 4 last year, one of the changes they made was a completely re-written software iSCSI initiator. This was done to optimize performance which is great considering how popular iSCSI SANs have become. They also gave the ability to use Round Robin MPIO (mutlipathing) in the software initiator in addition to Fixed Path and MRU which were previously available.

I’m working on a vSphere implementation using Dell EqualLogic SANs and wanted to configure Round Robin on all of my datastores. Dell has a great whitepaper on how to set this up, but unfortunately the document fails to mention one key thing: this doesn’t change the default path selection plugin (PSP) from Fixed to Round Robin.   That means that you’ll have to set the multipathing policy to Round Robin on all of your existing datastores and will have to remember to do that on all future datastores. When you’ve got multiple ESX hosts with lots of  datastores this can quickly become a pain.

Luckily there is a way to force the default multipathing policy to Round Robin. The following commands can be used to change the default PSP to Round Robin as well as configure round robin specifically for the EqualLogic provider.  These commands can be entered at the Service Console or via the vSphere CLI 4.0:

esxcli nmp satp setdefaultpsp –satp VMW_SATP_DEFAULT_AA –psp VMW_PSP_RR
esxcli nmp satp setdefaultpsp –satp VMW_SATP_EQL –psp VMW_PSP_RR
esxcli corestorage claimrule load
esxcli corestorage claimrule run

Note that “satp” and “psp” are preceded by two dashes and not a single dash as it appears in this blog post.

Once you enter those commands (no rebooting required) any volume you add, either new or existing, will use Round Robin MPIO by default.

A common situation in organizations is to make calendars public, so that employees can see other employee’s availability, and collaborate better.  Users may also delegate rights to other users to view their messages, tasks, and contacts.  In these situations, people may rely on marking sensitive items private to hide them from other users.  In Outlook or OWA, other users will see a placeholder for the private items, but won’t be able to view any of the details.  However, you should keep in mind that this privacy is only a feature of the client application–Outlook or OWA–and is not inherent to Exchange.  Exchange itself does not support any kind of item-level security or privacy, and only has a field called “sensitivity” which is used by Outlook and OWA.  The client applications look at that field to determine whether to display the item.

This architecture is common to all versions of Outlook and Exchange, through 2010, and is not really a bug, so much as an architectural decision by Microsoft to keep item-level permissions in the client-tier.  The end result is that people should realize that just because they mark an appointment or other item private in Outlook, it doesn’t mean that no one else will be able to see it.  Items that are extrememly sensitive should probably not be stored in Exchange in the first place, or you should take off all delegate / view rights to your mailbox.  People who you give delegate rights to should also be people who you trust.

Developers should note that when writing custom applications with WebDAV, Exchange Web Services, or any other method, all items will be returned including private items.  The custom application should look at the sensitivity setting of each message before displaying it.  If the sensitivity is private, then the mesage should not be displayed.  We, at Kraft Kennedy, have run into this issue several times when creating custom applications with WebDAV that pull back appointments from the Exchange calendar.

Microsoft has details about allowing other users to manage your mail and calendar here: 

http://office.microsoft.com/en-us/outlook/HA100750811033.aspx?pid=CH100788801033

Note, the last paragraph of the article:

I’ve seen a lot of talk lately about VMware’s Transparent Page Sharing (TPS) and how it is affected by ASLR in Windows 2008/Windows 7. I wanted to see if there was any real measurable reduction in shared memory when using ASLR vs. when it was disabled. First, let’s talk about what TPS and ASLR actually are and what the acronyms mean.

Transparent Page Sharing is a technology built into ESX/ESXi that looks for identical guest memory pages and writes them to memory just once. Guests can then share those identical pages rather than each writing the same page to memory. TPS is a great feature that allows for memory overcommittment, especially on hosts that run many of the same type of workload.

Address Space Layout Randomization (ASLR) is a security feature that randomizes the position of data in memory, making it more difficult for attackers to predict where data can be found while in memory. This feature has been enabled in Windows since Windows Vista, and other operating system such as Linux and MacOS implement this in some form as well.

Since ASLR randomizes information in memory it makes sense that it would be more difficult for TPS to find identical memory pages and thus memory sharing would be reduced. But just how much of a difference does it make? I decided to try and find out. Here are the specs from my test environment:

Server: HP DL385 G1 (AMD Opteron 275)
ESX: 4.0.0 build 244038
Guest OS: Windows Server 2008 R2
Guest RAM: 2.5GB

All guests were cloned from the same template and have the same software installed. On guests TESTSRV1 and TESTSRV3, I left the default settings. On TESTSRV2 and TESTSRV4, I disabled ASLR using the following regkey:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management]
“MoveImages”=dword:00000000

In all of my testing, including leaving VMs idle and also running memory tests, I found no measurable difference in the amount of memory shared with TPS. I also looked at esxtop to see how much memory was actually being shared and I saw virtually no difference whatsoever between VMs that had ASLR enabled and those that had it disabled.

Host Memory Usage:

esxtop statistics:

The SHRD and SHRDSVD columns represent how much memory is being shared with TPS and the total memory savings. Clearly there is quite a bit of memory sharing going with or without ASLR enabled.

Why would this be the case since it makes sense that TPS would be hurt by ASLR? ASLR requires applications to “opt-in” to have their memory randomized, and I suspect that much of Windows 2008 R2 is not opted in. Perhaps applications will come out in the future that are written to take advantage of ASLR, but at the moment that doesn’t appear to be the case.

Of course this is by no means a definitive test as it wasn’t run with production systems and real users running real applications. That said, I think it shows that ASLR does not dramatically reduce the amount of memory shared with TPS. I did also look at production systems left at the default settings (ASLR enabled) and saw similar memory sharing gains. I’m curious if others have seen similar results in their environments, so drop me a line if you’ve done any similar testing.

More info:

What is ASLR (Wikipedia)
Interpreting esxtop statistics