Issues after restoring MOSS to a previous VM snapshot

We restored our VM MOSS farm this morning from a snapshot taken a couple of months ago. Various issues occurred. Please note i am still working through some of the errors and will update this post as i find the answers.

1. I could no longer login to the server because the snapshot was older than 30 days - the computer account for the server had expired and so needed to be reset and reconnected to the domain. See this blog for steps to fix this.
   
2. The MOSS environment although appearing correct in Central Administration was not configured in IIS. The Central Administration site was the only site appearing in IIS even though all the web applications were still there.
   
   To fix this:


1. Delete the old web application using Central Administration.
2. Delete all the SSP web apps & DBs, apart from the default SSP which cannot be deleted.
3. Recreate the web applications with Temp DB names.
4. Recreate the SSPs with new DBs.
5. Change the default SSP to one of the newly created SSPs, then delete the original default SSP inc DB, then recreate it's web app and set back to the default updating any associations as required.
   
   At this point I ran in to some issues recreating the SSP with the following errors. First of all the SSP failed to provision with the error:
   
   Provisioning failed: A transport-level error has occurred when sending the request to the server.
   
   Looking in the Event Log i found the following related errors:
   
   Event ID 5554
   Failure during sweep synch. Exception was A transport-level error has occurred when sending the request to the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.).
   
   Event ID 7888
   A runtime exception was detected. Details follow.
   Message: A transport-level error has occurred when sending the request to the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
   
   Event ID 5586
   Unknown SQL Exception 10054 occured. Additional error information from SQL Server is included below.
   
   A transport-level error has occurred when sending the request to the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
   
   Solution:
   http://social.msdn.microsoft.com/forums/en-US/sqldatabaseengine/thread/0671c03b-5488-4be4-bc5a-579849fa0950
   Reboot the server and kill any remaining connections (delete the Admin accounts from local users & groups, reboot, add the Admin accounts back in).
   
   Also in the event log were the following errors:
   
   Event ID 10016
   The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID
   {3D42CCB1-4665-4620-92A3-478F47389230}
   
   Solution:
   http://sharemypoint.wordpress.com/2007/12/18/error-event-id-6398-and-6482-about-security-rights-of-osearch-service/
   Add the WSS_WPG, WSS_ADMIN_WPG, Search & Admin accounts to the OSearch DCOM Service
   
   Event ID 6141
   The site /ssp/admin could not be created. The following exception occured: This page has encountered a critical error. Contact your system administrator if this problem persists.
   
   Event ID 6610
   Safe mode did not start successfully. This page has encountered a critical error. Contact your system administrator if this problem persists.
   
   Event ID 5629
   Failed to load the SafeControl assembly paths for web.config. C:\Inetpub\wwwroot\wss\VirtualDirectories\ssp180
   
   Error importing WebPart. Assembly Microsoft.Office.Server.Search, Version=OAssemblyAssemblyVer, Culture=neutral, PublicKeyToken=OAssemblyPublicKey, TypeName. Microsoft.Office.Server.Search.WebControls.ActiveCrawls
   
   
1. I deleted the SSP web apps again.
   
2. Stop the search query & index services.
3. Start the search query & index services.
4. Restart IIS.
5. Recreate the SSP web app.
6. Recreate the SSP & set back to default with any associations required.
   
   Success - the SSP provisioned correctly.
   
   
1. Restore the DB from by opening the Content Database page in Central Administration. Delete the Temp DB then in STSADM associate the original DB:
   
   cd C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN
   
   stsadm.exe -o addcontentdb -url http://portal -databasename MOSS_MyDB_WSS_Content -databaseserver MyServer
   
   
3. If you receive the error 'An unexpected error has occured' open the web.config and update as follows:
   http://blog.thekid.me.uk/archive/2007/02/15/a-solution-to-quot-an-unexpected-error-has-occurred-quot-in-wss-v3.aspx
   
1. Change <SafeMode MaxControls=“200“ CallStack=“false“ to <SafeMode MaxControls=“200“ CallStack=“true“…>
2. Set custom errors to 'Off' <customErrors mode=“Off“/>

Changing MOSS service accounts & passwords

I have been preparing some servers for a bunch of testing with PerformancePoint Server (more to follow on this later) which required me to change our MOSS service accounts. Reading around the steps to update these credentials appeared pretty straight forward but as usual MOSS' idiosyncrasies were all to apparent. Here's what i did and how i managed to solve the issues that arose. Hope it helps someone else with the same challenge.

The accounts to be changed were as follows:

1. old_moss_admin ----> new_moss_admin
2. old_ssp_ap ---->new_ssp_ap
3. old_ap1_process ---->new_ap1_process

I followed the following MS KB instructions: 'How to change service accounts and service account passwords in SharePoint Server 2007 and in Windows SharePoint Services 3.0'

1. MOSS Admin Account Update

To update the account old_moss_admin to new_moss_admin:

1. Update the password for the account that is used by the Central Administration application pool. To do this, follow these steps:


1. On all servers in the server farm, open a command prompt, type the following line, and then press ENTER:
   cd %commonprogramfiles%\Microsoft Shared\Web server extensions\12\Bin


2. On the server that hosts the Central Administration Web site, type the following line at the command prompt, and then press ENTER:
   stsadm -o updatefarmcredentials -userlogin DomainName\UserName -password NewPassword


3. On all other servers in the server farm, type the following line at the command prompt, and then press ENTER:
   stsadm -o updatefarmcredentials -userlogin DomainName\UserName -password NewPassword -local


4. Restart Microsoft Internet Information Services (IIS) 6.0. To do this, type the following line at the command prompt, and then press ENTER:
   iisreset /noforce


2. Verify that the Administration Application Pool Credential Deployment job definition is no longer displayed on the Timer Job Definitions page of SharePoint 3.0 Central Administration. To do this, follow these steps:


1. Open SharePoint 3.0 Central Administration, click Operations, and then click Timer job definitions under Global Configuration.


2. Verify that the Administration Application Pool Credential Deployment job definition is no longer displayed in the list.

Error: My Central Administration is now no longer available with the error: 'Service Unavailable'

• I checked & retyped the password in CA web app pool.
• I checked the WSS_WPG, WSS_RESTRICTED_WPG and WSS_ADMIN_WPG local groups on each of the servers - all seemed to be up to date.
• Restarting the CA web app pool makes no difference.

At this point i rolled back to the previous credentials. I then tried again from the start following the same steps above. Result - my Central Administration was still unavailable with the error: 'Service Unavailable'

• I checked the IIS_WPG local group on each server. The new_moss_admin account was not updated so i added it to the group.

I then discovered a couple of DCOM errors in the System Event viewer.

1. I updated the new CA account 'new_moss_admin' to:


• IIS Admin Service
• IIS WAMREG Admin Service


2. Restarted IIS.

Success: Central Admin is up and running again.

2. SSP Account Update

To update the SSP account old_ssp_ap to new_ssp_ap:

Attempt 1 - The wrong way...

1. Open Central Administration
2. Navigate to the Operations page
3. Select Service Accounts
4. Select the radio button Web app pool
5. Select the web service - WSS Web App
6. Select the App Pool - SSP1
7. Select the radio button Configurable
8. Input the new credentials
9. Ok
10. Restarted IIS on both my servers

Error: This had no effect other than stopping me from accessing the SSP. Checking in IIS the account was not updated to the new credentials. I then tried to update the accout via IIS but this had noeffect either and on opening the App Pool properties again i discovered that the account credentials had changed themselves back to the old account.

Attempt 2 - The right way...

I then tried a different approach:

1. Open Central Administration
2. Navigate to the Shared Services Aministration homepage
3. Hover over SSP1's title and select Edit Properties from the drop down.
4. Scroll down to SSP Server Credentials
5. Delete the old credentials and input the new account username & password
6. Ok
7. I then changed the SSP to be the default SSP.
8. Open Compnent Services and update the DCOM services IIS Admin & IIS WAMREG with the new account.
   
9. Restarted IIS on both my servers.

Checking IIS the account was still not updated to the new credentials and i still could not access the SSP website.

Success: 10 minutes later having searched on Google again and felt thoroughly annoyed at there being no pages I hadn't yet read on this topic I checked the SSP website again and it worked! Checking IIS the accounts have now updated to the new credentials.

At this stage I discovered some new errors in the Event View

1. Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (5d77cabf-3414-40cc-a5ef-f30226a9288b).
   
   Reason: The program attempted to update an object that was updated by another user.
   
   
2. Eventid 6482 - Reason: Access to the path 'C:\WINDOWS\system32\drivers\etc\HOSTS' is denied.
   
   This error was now occuring every minute.
   
   The solution to this (and the previous error) can be found in my previous post here: http://bytelab.blogspot.com/2008/02/sharepoint-search-service-cannot-find.html
   

3. Application Pool Process Accout Update

To update the web app pool process account old_ap1_process to new_ap1_process:

1. Open Central Administration
2. Navigate to the Operations page
3. Select Service Accounts
4. Select the radio button Web app pool
5. Select the web service - WSS Web App
6. Select the App Pool - MOSS Portal
7. Select the radio button Configurable
8. Input the new credentials
9. Ok
10. Open Compnent Services and update the DCOM services IIS Admin & IIS WAMREG with the new account.
11. Restarted IIS on both my servers

Success: All updated and working ok.

************************************************************

Useful web sites i visited for this:

• Change passwords for Central Administration and Timer service (Windows SharePoint Services) http://technet.microsoft.com/en-us/library/cc288991.aspx
• How to change service accounts and service account passwords in SharePoint Server 2007 and in Windows SharePoint Services 3.0 http://support.microsoft.com/kb/934838
• MOSS 2007 Setup Accounts Clayton J http://claytonj.wordpress.com/2007/04/23/moss-2007-setup-accounts/
• How to change service accounts and their passwords in MOSS and WSS 3.0 http://weblogs.asp.net/erobillard/archive/2007/07/06/how-to-change-service-accounts-and-their-passwords-in-moss-and-wss-3-0.aspx
  

SSP Stuck on Unprovisioning

I tried to delete a test SSP the other day via Central Administration. Having selected the options to also delete associated content databases and web applications i hit ok and left it to run. The SSP status changed to 'Unprovisioning' and an hour later was still stuck.

I then followed this example in the hope the SSP would delete.

• http://prequest01.wordpress.com/2008/08/16/unable-to-delete-shared-services/
  
  
1. Tried: stsadm -o deletessp "TestSSP" –force, but this did not work
   
2. Login to SQL server.
   
3. Open SQL Management Studio and expend Databases.
   
4. Expand Configuration Database & Tables.
   
5. Opened table for dbo.object.
   
6. Executed following query in query analyzer: SELECT * FROM [MOSS_CFG_CA_01].[dbo].[Objects]where name like ‘Name of the Shared Services’.
7. Copy the ID of object referenced in objects table of configuration database.
   
8. Open command prompt and changed directory to C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN> and executed following command to delete the Shared Services using the ID which was copied: Stsadm -o deleteconfigurationobject -id “id retrieved from object table”

• To clean up we decide to delete the DBs via SQL Management Studio and delete the web applications via Central Administration.

A week later I discovered that one of our servers had rebooted itself with the following log (don't yet know if this is related):

System Failure: Stop error
Reason Code: 0x805000f
Bug ID:
Bugcheck String: 0x000000c5 (0x00000004, 0xd0000002, 0x00000001, 0x8089bce3)
Comment: 0x000000c5 (0x00000004, 0xd0000002, 0x00000001, 0x8089bce3)

I then discovered that the search was no longer working. Submiting a query returned the following error:

Cannot connect to the search service

The status of the test SSP i had tried to delete was still stuck on 'Unprovisioning'. Also the other SSPs 'Edit Properties' pages in Central Administration were no longer available and returned the following error:

An unhandled exception occurred in the user interface.Exception Information: Cannot open database "xxx" requested by the login. The login failed. Login failed for user 'xxx'.

In the event log on the server I found multiple instances of the following error:

Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (e6a1ecb4-3aa8-45cd-a179-d7e053aa927a).

Reason: Cannot open database "xxx" requested by the login. The login failed.
Login failed for user 'xxx'.

Solution:

1. Restart the Office SharePoint Server Search service in via the Computer Management console - Search still not working.
2. I retried the STSADM CMD
   stsadm -o deletessp "TestSSP" –force

Operation was successful this time and the SSP has now gone. The event log is now clear of all errors and the unavailable pages in Central Administration are now available again.

Duplicate Site ID Error in the Application Event Log

After rebuilding our farm environment onto a new server with SP1 we started receiving these errors on the hour every hour. It relates to a database that was renamed and thus disassocoated with its web application, then reassociated with it using STSADM -addcontentdb.

Failure trying to synch web application d9ba0571-84d0-40c1-add9-3be5a5a5a443, ContentDB 4606754f-fb82-4829-bda1-878869c2a450 Exception message was A duplicate site ID 18318f24-21f6-4b45-95c3-5479fe6f99dc(http://ssp4dev/ssp/admin) was found. This might be caused by restoring a content database from one server farm into a different server farm without first removing the original database and then running stsadm -o preparetomove. If this is the cause, the stsadm -o preparetomove command can be used with the -OldContentDB command line option to resolve this issue.

The solution

stsadm -o sync -DeleteOldDatabases 0

Refer to MS for the full STSADM sync command info.

stsadm -o sync

-excludewebapps
-synctiming
-sweeptiming
-listolddatabases
-deleteolddatabases

STSADM Associate Content Database

C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN>stsa
dm.exe -o addcontentdb -url http://21cndev -databasename SHAREPOINT_21CN_WSS_Con
tent_Temp -databaseserver tdcssqlsp02

Customizing task alert email notifications in MOSS using STSADM

stsadm -o updatealerttemplates -url http://server -filename "C:\Program files\common files\Microsoft shared\web server extensions\12\template\XML\alerttemplates.xml" stsadm -o updatealerttemplates -url http://server -filename "C:\Program files\common files\Microsoft shared\web server extensions\12\template\XML\alerttemplates.xml"

Enumssptimerjobs: STSADM operation (Office SharePoint Server)

The primary purpose of this operation is for an administrator to list all timer jobs in the SSP especially if a computer had Microsoft Project Server installed, and then was uninstalled. In this scenario, the Project Server SSP timer jobs still exist in the SSP database even though Project Server is not installed on the computer. To delete these orphan timer jobs, use the Deletessptimerjob operation. For more information about timer jobs, see the Remarks section of this article.

Syntax

stsadm -o enumssptimerjobs

-title

'stsadm -o canceldeployment' when a solution deployment is already under way

If a SharePoint solution deployment fails, yet the timer job is created and not running, you will be unable to redeploy or retract the solution. The retract / deploy buttons will not be present in Central Admin and stsadm will declare that a "deployment or retraction is already under way for the solution". You can find yourself in this state if the “Windows SharePoint Services Administration” service is stopped when you first attempt to deploy, but even after you realise your mistake and start that service, the solution deployment is still stuck.
The answer is hidden in that long list of stsadm operations:

stsadm –o canceldeployment –id

This command will cancel the timer job that is meant to deploy the solution. Once it has been cleared, you can deploy successfully. Obtain the id parameter from the url to the appropriate deployment timer job in the Operations -> Timer Job Definitions page

Deploying Web Parts using CAB files

1. For ease of deployment, copy the .cab file to the following location, where Office SharePoint Server 2007 is installed:
   

   C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\bin

   This should be the same as the location of Stsadm.exe. Ensure that you update the path on your computer if it does not match the default location.
   


2. Click Start, click Run, and then type cmd.
   


3. Type cd "C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\bin"
   


4. Run the following command to deploy your Web Part accross the entire farm:
   
   stsadm.exe -o addwppack -filename yourcabfilename.cab -globalinstall -force
   


5. Run the following command to deploy your Web Part to a specific website:
   
   stsadm.exe -o addwppack -filename yourcabfilename.cab -url http://teleport -globalinstall -force
   


6. After the command returns with "Operation completed successfully," run the following command:
   
   iisreset /timeout:0
   


7. Check central administration / solution management to see where the solutions are registered as deployed by SharePoint.
   

Creating a Custom Master Page & Deploying it as a Feature

First of all thanks to Heather Soloman who posted a great article about this Create a Feature: Master Pages for Site Collections.

Here's how i set up the Custom Masterpages and depoyed them as a feature:

Create the Feature

1. Create a folder called CustomMasterPages under the Features directory on the web server.
2. Inside the CustomMasterPages folder create the folders:
   
   • en-us
   • Images
   • Styles
   • MasterPages
   • PageLayouts
   • JavaScript
3. Copy these two files from the Features directory PublishingLayouts:
   
   • Feature.xml, and
   • ProvisionedFiles.xml
4. Create a file in the MasterPages folder called CustomMaster.master
5. Open CustomMaster.master in notepad and paste the following minimal master page code from Microsoft.
6. Alternatively Heather Soloman has created another version of the minimal master page which is the version I used in the end. The reason for this is that Heather's version is a stripped out version of the default.master master page which includes all the functionality available. The Microsoft version has only the minimal elements included.
7. Format the master page with XHTML tags & CSS.
8. Change the doc type to XHTML transitional.
   
   <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/transitional.dtd">
   


9. Edit the ProvisionedFiles.xml file to provision the master page, CSS files & images etc.
   
   
   <!-- _lcid="1033" _version="12.0.4407" _dal="1" -->
   <!-- _LocalBinding -->
   <Elements xmlns="http://schemas.microsoft.com/sharepoint/">
   
   <Module Name="OSGMasterPages" Url="_catalogs/masterpage" Path="MasterPages" RootWebOnly="TRUE">
   <File Url="tr_portalMaster.master" Type="GhostableInLibrary">
   <Property Name="ContentType" Value="Teleport Master Page - DEFAULT PORTAL LAYOUT" />
   <Property Name="PublishingPreviewImage" Value="~SiteCollection/_catalogs/masterpage/$Resources:core,Culture;/Preview Images/tr_teleportLogo100.gif, ~SiteCollection/_catalogs/masterpage/$Resources:core,Culture;/Preview Images/tr_teleportLogo100.gif" />
   <Property Name="MasterPageDescription" Value="This is the TelePort master page (DEFAULT CSS LAYOUT) for use with collaboration or publishing sites." />
   </File>
   </Module>
   <Module Name="PublishingLayoutsPreviewImages" Url="_catalogs/masterpage" IncludeFolders="??-??" Path="" RootWebOnly="TRUE">
   <File Url="tr_teleportLogo100.gif" Name="Preview Images/tr_teleportLogo100.gif" Type="GhostableInLibrary">
   </File>
   </Module>
   <Module Name="Images" Url="Style Library/Images" Path="Images" RootWebOnly="TRUE">
   <File Url="tr_telerealLogo102.gif" Name="tr_telerealLogo102.gif" Type="GhostableInLibrary" />
   </Module>
   <Module Name="OSGStyles" Url="Style Library" Path="Styles" RootWebOnly="TRUE">
   <File Url="tr_core.css" Type="GhostableInLibrary" />
   <File Url="tr_portalMaster.css" Type="GhostableInLibrary" />
   <File Url="tr_portalMaster_IE6.css" Type="GhostableInLibrary" />
   <File Url="tr_portalMaster_IE55.css" Type="GhostableInLibrary" />
   </Module>
   </Elements>
   
   
10. Add a preview image to the language folder (en-us).
11. Create a unique GUID
12. In the Feature.xml file:
    
    • Change the feature ID to the unique GUID
    • Change the title to Custom Master Pages
    • Change the Hidden state to False
      
13. Recycle the app pool on the WFE website

Deploying the Feature

1. On the web server (DEV02) launch command prompt and change dir to:
   
   cd C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN
   


2. Install the feature using STSADM:
   
   stsadm -o installfeature -name CustomMasterPages
   


3. Go to Site Collection Features on the portal top level site and activate the feature.
   
   To use the master page select from the drop down list on Site Settings / Site Master Page Settings

Redeploying a Feature

Changes made to the feature require that the feature is redeployed. For example; adding new master pages, css files etc. To do this:

1. On the web server (DEV02) launch command prompt and change dir to:
   
   cd C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN
   


2. Reinstall the feature using STSADM:
   
   stsadm -o installfeature -name CustomMasterPages -force
   


3. Reactivate the feature in Site Collection Features by deavctivating and the activating it.