NOLOCK Errors with CU3 Installed
We have had CU3 installed in our demo environment for several months without issue. A few days ago, we noticed some subscriptions were not working and that the Alert Views were either non-responsive, empty, or giving NOLOCK errors (Could not continue scan
with NOLOCK due to data movement). Tried reindexing the DB but no change. I'm wondering if it's possible there is larger corruption in the database and we need to restore, or if there is some other specific fix that could be applied post-CU3. Here is
the full error from the SDK service:
An exception was thrown while processing GetObjectsFromReader for session id uuid:005e9e67-365e-4541-98d8-cdf49a4c2e69;id=1.
Exception Message: The creator of this fault did not specify a Reason.
Full Exception: System.ServiceModel.FaultException`1[Microsoft.EnterpriseManagement.Common.UnknownDatabaseException]: The creator of this fault did not specify a Reason. (Fault Detail is equal to Could not continue scan with NOLOCK due to data
movement.).
November 5th, 2010 11:24am
has the rms or its services been bounced yet?Scott Moss MVP (Operations Manager) | President - System Center Virtual Users Group | Vice President - Atlanta Southeast Management Users Group (ATL SMUG) Please remember to click Mark as Answer on the post that helps you!
Free Windows Admin Tool Kit Click here and download it now
November 5th, 2010 11:47am
We bounced it a few times. I can also duplicate the NOLOCK errors with direct queries against the alertview tables in the operational DB.
November 5th, 2010 11:59am
ouch. unfortunately the only recent reference to a NOLOCK situation was suppost to have been fixed in CU3. So lets give this a bump and see if anyone else has something to add to this.
http://social.technet.microsoft.com/Forums/en-US/operationsmanagergeneral/thread/da8f6529-e645-4b68-83b7-f6fe9f190c15Scott Moss MVP (Operations Manager) | President - System Center Virtual Users Group | Vice President - Atlanta Southeast Management Users Group (ATL SMUG) Please remember to click Mark as Answer on the post that helps you!
Free Windows Admin Tool Kit Click here and download it now
November 6th, 2010 12:48am
As Scott mentions a NOLOCK issue should have been fixed in CU3, but if you can simulate di issue with a simple SQL query against the DB then you could have an alert storm running or you SQL Server could have performance problems. Please check the RMS OpsMgr
event log for any data access related errors and post here if any.- Daniele, Microsoft MVP OpsMgr This posting is provided "AS IS" with no warranties, and confers no rights. http://nocentdocent.wordpress.com http://www.progel.it
November 6th, 2010 5:36am
There aren't any data access related errors, but general bursts of unhappiness due to errors whenever something accesses the Alert table. On example of a query that works in our other OpsMgr environments but is bombing with NOLOCKs in our demo is this
simple alerts per day query:
SELECT CONVERT(VARCHAR(20), TimeAdded, 102) AS DayAdded, COUNT(*) AS NumAlertsPerDay
FROM Alert WITH (NOLOCK)
WHERE TimeRaised is not NULL
GROUP BY CONVERT(VARCHAR(20), TimeAdded, 102)
ORDER BY DayAdded DESC
If there was an alert storm, is there a safe way to scrub the entire Alert table (it's not production, so I don't need the data)? Or is it better to try restoring to a good known point in time?
Free Windows Admin Tool Kit Click here and download it now
November 8th, 2010 11:20am
It turns out that the Alert and AlertHistory table in the operational database had some corrupt pages. Our DBAs did a restore of those tables and cleaned them up and the errors went away. Not sure if an event storm caused the corruption or it was unrelated
to OpsMgr itself. Thanks!
November 8th, 2010 4:59pm