MOSS Search Broken

MSS search has been broken for a few weeks now. One of the issues with the fault has been that MOSS had stopped writing to the error logs so we had no way of finding specific error inforation. I logged a call with Microsoft because of this and they suggested that i recreate the SSPs as they must be coorupt.

I tested this by creatiing a new SSP and true enough it did allow search queries and returned results. But, recreating our SSPs is no small job because once you do so you have to recreate all your customisations as well. This means redeploying custom webparts and setting them up on your sites. Recreateing audiences and resetting targeted content on the sites, etc etc. Depending on your set up this can take quite an effort.

Before heading down the last resort of recreating the SSPs i decided to try and solve the issues first. For info I am running a standard installation with 3 servers:

• MOSS01 - Index
• MOSS02 - WFE
• MOSS03 - Query & Central Admin

Take note that it took a weekend of digging around and trying the same steps over and over again to finally fix everything. There's a list of the errors and how to fix them at the bottom of this post. I have tried to put the main steps that i followed into order below - i hope this helps someone...

******************************************************

Attempt 1

1. Reboot Index server
2. Reboot Query server
   
3. Propagate the Index file to a new location using STSADM
   stsadm -o osearch -propagationlocation

Result: Index moved to new location sucessfully. Search not working.

Attempt 2

1. Start the Indexing service on the MOSS Index server using Computer Management / Services
   
2. Stop the Search Services (Index & Query) using STSADM
   stsadm -o osearch -action stop
3. Start the Search Services (Index & Query) using STSADM
   stsadm -o osearch -action start

Result: Search not working.

Attempt 3

1. Reset the Indexer in each SSP to the Index server via Central Admin SSP Proerties page
2. Add the process accounts to each SSP on the Edit Properties page - http://ssp/admin/_layouts/searchsspsettings.aspx
3. Set the security for C:\Windows\Tasks to read & write for the WSS_WPG account:
1. On the Index server make sure that you can see the Sharing and Security tab in the Windows tasks folder (usually C:\Windows\Tasks) by open a command prompt and type attrib –s %windir% \tasks.
2. Browse to C:\Windows\Tasks in explorer, right click and select properties. Grant the WSS_WPG group Read and Write permissions on the tasks folder.
3. Open a command prompt and type attrib +s %windir% \tasks to reset the tasks folder to its default view.
4. Delete the content index in each SSP
5. Start a full crawl in each SSP

Result: Search now returning results correctly in 2/4 SSPs. MOSS event logs are now populating.

• SSP1 status - Propagating to new Query server. Search index status 'Computing ranking'. Search not working. When I manually started a crawl, there's a pop up message saying 'Crawling might be paused because a backup or an index move operation is in progress. Are you sure you want to resume this crawl?' Selected 'Yes'.
  
• SSP2 status - Propagating to new Query server - Search not working
• SSP3 status - Propagating - Search is working
• SSP4 status - Idle - Search is working
  

Attempt 4

1. Reboot Index & Query servers
   
2. Stop the Search Services (Index & Query) using Central Administration
   
3. Recreate the Search Services (Index & Query) using Central Administration
   
4. Reset the Indexer in each SSP to the Index server
5. Set the security for C:\Windows\Tasks to read & write for the WSS_WPG account
6. Reset the content index in each SSP
7. Start a full crawl in each SSP

Result: Search working in all SSPs.

One remaining error in the event log is now: Tracing Service failed to create the trace log file directory 'D:\MOSS\12\LOGS'. Error 3: The system cannot find the path specified.

******************************************************

General Error List

Some of the errors i have worked through are listed here:

• Your search cannot be completed because of a service error. Try your search again or contact your administrator for more information
• Reassociating the indexer and search service with the ssp under application management on the central administration site
  
• Content index on Portal_Content could not be initialized. Error The content index is corrupt.
• Query server not responding
• http://brijesh.spaces.live.com/blog/cns!BFBD772FBDA58C6D!194.entry
1. stsadm.exe –o osearch –propagationlocation index file path
2. Run a full crawl
   
• Could not create a database session
• http://social.technet.microsoft.com/Forums/en-US/sharepointsearch/thread/e277cc4f-03d9-4424-af1a-a167fd3c6186/
1. Open Central Admin
2. In the Application Tab go to Create or Manage this farm's shared services
3. Click the drop down of the SSP you want to edit, and click edit
4. Add the Indexing server
• Error trying to access the SSP search settings page: http://ssp/admin/_layouts/searchsspsettings.aspx - 403 FORBIDDEN
  
• See next solution below http://support.microsoft.com/kb/926959
• Error when you try to edit the content source schedule in Microsoft Office SharePoint Server 2007: "Access is denied"
• http://support.microsoft.com/kb/926959
  To work around this issue, you must add the WSS_WPG group to the Tasks folder. To do this, follow these steps:
1. Use an account that has administrative permissions to log on to the computer that is running the Office SharePoint Server 2007 indexing service.
2. Click Start, click Run, type cmd, and then click OK.
3. At the command prompt, type the following command, and then press ENTER:
   attrib –s %windir%\tasks
   Note In this example, %windir% is the path of the Windows folder. For example, the path can be C:\Windows.
   Note If Windows Explorer is open when you make this change, you will not see the extra tab in Windows Explorer. If Windows Explorer is already open, close and then reopen it before you perform step 4.
4. In Windows Explorer, right-click the Tasks folder, and then click Properties.
5. In the Tasks Properties dialog box, click the Security tab, and then click Add
6. In the Select Users, Computers, or Groups dialog box, type WSS_WPG in the Enter object names to select box, and then click OK.
7. Grant the following permissions for the WSS_WPG account, and then click OK: Read & Write
8. Click Start, point to Administrative Tools, and then click Internet Information Services (IIS) Manager.
9. In Internet Information Services (IIS) Manager, right-click ComputerName (local computer), click All Tasks, and then click Restart IIS.
10. Click Start, click Run, type cmd, and then click OK.
11. At the command prompt, type the following command, and then press ENTER:
    attrib +s %windir%\tasks
    Note This resets the Tasks Property back to the default view.
    
• The search service is currently offline. Visit the Services on Server page in SharePoint Central Administration to verify whether the service is enabled. This might also be because an indexer move is in progress.
• http://www.sharepointblogs.com/jasonmedero/archive/2008/09/15/moss-2007-error-when-moving-indexing-role-the-search-service-is-currently-offline-visit-the-services-on-server-page.aspx
1. Stop and start search services on new index server and all servers running the query role
2. Then restarted the search service via central administration under operations tab>>Services
3. Went into Services.msc and found the Office Search Service and restarted it manually
4. Checked to make sure that the search service account had correct rights to SQL.
• None of the above worked so...
  
1. Going into the SSP settings (clicking on SSP administration not the SSP link itself) within the left quick launch navigation from within central administration.
2. Selecting the dropdown arrow next to the SSP where I was having the search issue
3. Select edit properties
4. Within the properties screen of the SSP look for the “Process accounts”
5. Make sure that your search account is in there!
   
• Test Shared Service Provider stuck on 'Unprovisioning' after trying to delete it and associated databases.
• http://www.synergyonline.com/blog/blog-moss/Lists/Posts/Post.aspx?ID=19
• Tried: stsadm -o deletessp –force, but this did not work
  
• http://prequest01.wordpress.com/2008/08/16/unable-to-delete-shared-services/
1. Login to SQL server.
   
2. Open SQL Management Studio and expend Databases.
   
3. Expand Configuration Database & Tables.
   
4. Opened table for dbo.object.
   
5. Executed following query in query analyzer: SELECT * FROM [MOSS_CFG_CA_01].[dbo].[Objects]where name like ‘Name of the Shared Services’.
6. Copy the ID of object referenced in objects table of configuration database.
   
7. Open command prompt and changed directory to C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN> and executed following command to delete the Shared Services using the ID which was copied: Stsadm -o deleteconfigurationobject -id “id retrieved from object table”
• The search request was unable to connect to the Search Service
• http://anuraj.wordpress.com/2007/10/25/the-search-request-was-unable-to-connect-to-the-search-service/
1. Start the Indexing service, which is disabled by default.
2. Go to Services from Control Panel > Administrative Tools > Services
   
3. Select the Indexing service, right click and choose the option start