• The Autonomy Control Center - a web app intended to simplify management of the IDOL components, particularly in a multi-server environment
• Changes to the Active Content engine that improve query performance
• Improvements to the Deployment Tool that allow more granular control when deploying various IDOL components
• A shift of Eduction processing from the IndexTask component in IDOL to the WorkSite Ingestion Service
• Automatic restarting of the WorkSite Connector during the initial crawl if the service dies before the first crawl is complete

These, as well as a full list of resolved issues and enhancements, can be found in more detail in the “WorkSite Indexer Release Notes (8.5 SP2)” on Autonomy’s support website.  We’ve been excited to get our hands on the new releases and have been working on deploying them over the past few days.  The WorkSite Indexer has thus far been deploying without any trouble.

The Universal Search server, on the other hand, has proven to be a little more difficult.  We’ve discovered a bug (confirmed by Autonomy) with deploying multiple WorkSite content engines that causes the deployment process to stop with the message “null” followed by “Deployment process incomplete”, as shown below:

Short of attempting to deploy with 1 WorkSite content engine and creating the additional content engines manually, this brings the deployment of IUS 8.5 SP2 to a screeching halt.  Autonomy has identified this as a high priority issue and expects to have a resolution out in the very near future.  We’ll be keeping our eyes out for the updated release, and will provide a more in-depth review as we spend some time with the new features.  Stay tuned…

Using the 8.5 SP1 (and above) Deploy Tool, you can now choose whether to include the Active Content Engine (ACE) in your deployment.  Nice improvement over the first iteration where the entire ACE implementation was a manual process.   There’s also an option to perform an Initial Indexing procedure.  From the Indexer Administrator’s Guide:

Initial Indexing- Select this option to build a pre-configured deployment package for the initial build of the index. With this setting enabled, the WorkSite Indexer is optimized to index a large dataset quickly; the Active Content engine is not used with this option enabled. This option is strongly recommended to decrease the time it takes for WorkSite Indexer to complete the initial index build. After the initial index has been created, deselect this option, regenerate the configuration files and apply them to run WorkSite Indexer in normal mode.

The thing to be aware of with this option is that documents weren’t committed until the entire crawl was complete.  So users were unable to get full text results during the initial build –  much like the first release of the 8.3 Indexer (driven by the Vivisimo engine).  If you are building this index in parallel with a production indexer, then users aren’t affected.

The other thing for you to keep in mind is that even though 8.5 SP1 P1 was released after the 8.5 SP1 Incremental Update, you still need to apply the Incremental Update to your environment.  Autonomy has informed me that the Incremental Update will be included in a P2.   No word on when that is to be expected, however.

In summary, the 8.5 SP1 P1 install and deployment is much slicker and automated than the previous 8.5 IDOL indexer releases.  The integration of the ACE into the Deploy Tool is a nice feature.  The Initial Index feature is nice, as long as you know the potential user impact during the crawl.

In summary, there are 3 types of items that need to be backed up, and then restored:

1. All WorkSite Indexer configuration files (*.cfg)
2. All WorkSite Connector files (*.db)
3. All WorkSite Content index backups (the actual indices themselves)

The first two can be configured as simple robocopies of the production *.cfg and *.db files into a similar backup folder structure.    The third can be scheduled by updating each WorkSite Content *.cfg file to include particular options in the [Schedule] section.  The key thing to remember about the WorkSite Content is that the backup will be just about as large as your production WorkSite Content, so you would want to either make sure there is enough local hard drive space or available space on an accessible SAN LUN to store the backup copy as it is put onto tape.  This is something that needs to be considered when estimating the storage requirements of the system.

To restore the index from the backup, you need to ensure that the *.cfg files are in place.  If they are not, then copy these from your backup location into the appropriate production folders.  The next step is to restore the appropriate WorkSite Connector *.db files.     These aren’t really database files, but rather a timestamp to tell IDOL from which document modification date to start rebuilding the index.  For example, indexing all documents that were modified after 5pm last Tuesday.   Unfortunately, the timestamp looks like gibberish to the human eye, if you ever have to modify it.   From the Administrator’s Guide:

If no db file back up exists from the same time as the index data back up, the timestamp of the Connector db file can be manually modified to coincide with the time of the index data to be restored.

The WorkSite Connector db time should always be set to a time immediately before the index data back up in order to ensure that the WorkSite Connector begins indexing from a time that is synchronized with the index data. Setting the date time too early can result in unnecessary re-indexing and therefore impact disk space as well as performance; setting the date time too late can result in a gap of new content and edits that is not indexed.

For reference, the timestamp value stored in the WorkSite Connector db file is in EPOCH format (time since 12:00 a.m. January 1, 1970 in milliseconds). For example, a timestamp of “1256942003000″ in the file translates to “22:33 UTC October 30th, 2009″ in a standard date notation.

Figuring out how many milliseconds after midnight on January 1, 1970?  That sounds pretty daunting.  But there are conversion tools available online to help with this.     After each *.db file is in place, each WorkSite Content engine would need to be restored using the /DREINITIAL command pointing to the backed up Index.  

Further details are available in the documentation, but the key takeaway is to avoid using Backup Exec or a SAN snapshot and expect a healthy restored index.  There are special steps that need to be taken to ensure a reliable backup.