HealthService.exe 100% CPU
We have been seeing 100% CPU usage by the HealthService.exe, I watched this process using Procmon and it seems to be when the HealthService is reading in the management pack XML files in C:\Program Files\System Center Operations Manager 2007\Health Service
State\Management Packs\ for every 2k read of the XML file it is doing 40 registry operations, for some manangement packs (Exchange and ISA) these XML files are over 10MB in size, which is a lot of registry operations.
The reg keys/values being accessed are as follows:-
241245 14:16:37.8303590 HealthService.exe 10336 ReadFile C:\Program Files\System Center Operations Manager 2007\Health Service State\Management Packs\Microsoft.Exchange.2007.{4BF99177-714F-AA15-C59A-ABD5C15FDC78}.{7D625363-7C19-B887-5A4A-EA4B86265DC6}.xml SUCCESS Offset:
9,496,948, Length: 2,038
241246 14:16:37.8304162 HealthService.exe 10336 RegOpenKey HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001 SUCCESS
241247 14:16:37.8304334 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001\Name SUCCESS Type: REG_SZ, Length: 80, Data: Microsoft Strong Cryptographic Provider
241248 14:16:37.8304436 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001\Name SUCCESS Type: REG_SZ, Length: 80, Data: Microsoft Strong Cryptographic Provider
241249 14:16:37.8304541 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001\Name SUCCESS Type: REG_SZ, Length: 80, Data: Microsoft Strong Cryptographic Provider
241250 14:16:37.8304639 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001\Name SUCCESS Type: REG_SZ, Length: 80, Data: Microsoft Strong Cryptographic Provider
241251 14:16:37.8304746 HealthService.exe 10336 RegCloseKey HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001 SUCCESS
241252 14:16:37.8304898 HealthService.exe 10336 RegOpenKey HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider SUCCESS
241253 14:16:37.8305059 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider\Type SUCCESS Type: REG_DWORD, Length: 4, Data: 1
241254 14:16:37.8305169 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider\Image Path SUCCESS Type: REG_SZ, Length: 22, Data: rsaenh.dll
241255 14:16:37.8305268 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider\Image Path SUCCESS Type: REG_SZ, Length: 22, Data: rsaenh.dll
241256 14:16:37.8305371 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider\Image Path SUCCESS Type: REG_SZ, Length: 22, Data: rsaenh.dll
241257 14:16:37.8305470 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider\Image Path SUCCESS Type: REG_SZ, Length: 22, Data: rsaenh.dll
241258 14:16:37.8305746 HealthService.exe 10336 RegOpenKey HKLM\Software\Microsoft\Cryptography SUCCESS
241259 14:16:37.8305893 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\MachineGuid SUCCESS Type: REG_SZ, Length: 74, Data: 18baea45-2456-4bf9-95da-637b783bcf1f
241260 14:16:37.8305986 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\MachineGuid SUCCESS Type: REG_SZ, Length: 74, Data: 18baea45-2456-4bf9-95da-637b783bcf1f
241261 14:16:37.8306080 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\MachineGuid SUCCESS Type: REG_SZ, Length: 74, Data: 18baea45-2456-4bf9-95da-637b783bcf1f
241262 14:16:37.8306170 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\MachineGuid SUCCESS Type: REG_SZ, Length: 74, Data: 18baea45-2456-4bf9-95da-637b783bcf1f
241263 14:16:37.8306295 HealthService.exe 10336 RegCloseKey HKLM\SOFTWARE\Microsoft\Cryptography SUCCESS
241264 14:16:37.8306448 HealthService.exe 10336 RegOpenKey HKLM\Software\Microsoft\Cryptography\Offload NAME NOT FOUND
241265 14:16:37.8306677 HealthService.exe 10336 RegCloseKey HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider SUCCESS
241266 14:16:37.8306977 HealthService.exe 10336 RegOpenKey HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001 SUCCESS
241267 14:16:37.8307143 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001\Name SUCCESS Type: REG_SZ, Length: 80, Data: Microsoft Strong Cryptographic Provider
241268 14:16:37.8307244 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001\Name SUCCESS Type: REG_SZ, Length: 80, Data: Microsoft Strong Cryptographic Provider
241269 14:16:37.8307350 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001\Name SUCCESS Type: REG_SZ, Length: 80, Data: Microsoft Strong Cryptographic Provider
241270 14:16:37.8307449 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001\Name SUCCESS Type: REG_SZ, Length: 80, Data: Microsoft Strong Cryptographic Provider
241271 14:16:37.8307557 HealthService.exe 10336 RegCloseKey HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider Types\Type 001 SUCCESS
241272 14:16:37.8307708 HealthService.exe 10336 RegOpenKey HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider SUCCESS
241273 14:16:37.8307871 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider\Type SUCCESS Type: REG_DWORD, Length: 4, Data: 1
241274 14:16:37.8307980 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider\Image Path SUCCESS Type: REG_SZ, Length: 22, Data: rsaenh.dll
241275 14:16:37.8308079 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider\Image Path SUCCESS Type: REG_SZ, Length: 22, Data: rsaenh.dll
241276 14:16:37.8308182 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider\Image Path SUCCESS Type: REG_SZ, Length: 22, Data: rsaenh.dll
241277 14:16:37.8308283 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider\Image Path SUCCESS Type: REG_SZ, Length: 22, Data: rsaenh.dll
241278 14:16:37.8308561 HealthService.exe 10336 RegOpenKey HKLM\Software\Microsoft\Cryptography SUCCESS
241279 14:16:37.8308709 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\MachineGuid SUCCESS Type: REG_SZ, Length: 74, Data: 18baea45-2456-4bf9-95da-637b783bcf1f
241280 14:16:37.8308801 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\MachineGuid SUCCESS Type: REG_SZ, Length: 74, Data: 18baea45-2456-4bf9-95da-637b783bcf1f
241281 14:16:37.8308895 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\MachineGuid SUCCESS Type: REG_SZ, Length: 74, Data: 18baea45-2456-4bf9-95da-637b783bcf1f
241282 14:16:37.8308986 HealthService.exe 10336 RegQueryValue HKLM\SOFTWARE\Microsoft\Cryptography\MachineGuid SUCCESS Type: REG_SZ, Length: 74, Data: 18baea45-2456-4bf9-95da-637b783bcf1f
241283 14:16:37.8309141 HealthService.exe 10336 RegCloseKey HKLM\SOFTWARE\Microsoft\Cryptography SUCCESS
241284 14:16:37.8309307 HealthService.exe 10336 RegOpenKey HKLM\Software\Microsoft\Cryptography\Offload NAME NOT FOUND
241285 14:16:37.8309542 HealthService.exe 10336 RegCloseKey HKLM\SOFTWARE\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider SUCCESS
Is there any reason for this much registry traffic just to read 2k of an XML file?
We are running SCOM 2007 (6.0.6278.0), it appears to hit the VMWare servers a lot harder than the physical servers. It happens on 2003/2008 servers, I have looked at the KB article for the MSXML hotfix but this is an older dll than we currently have installed.
July 14th, 2010 4:58pm
I would suggest installing the XML hotfix on one machine and seeing if that helps with the problem you are seeing.
Let us know if that doesnt help.Scott Moss MVP (Operations Manager) | President - System Center Virtual Users Group | Vice President - Atlanta Southeast Management Users Group (ATL SMUG)
Free Windows Admin Tool Kit Click here and download it now
July 14th, 2010 6:50pm
I would look elsewhere to diagnose the 100% CPU. Most windows programs read the registry - a lot. Windows does this very efficiently and the result will be negligible.
Look instead for which scripts are running - and which MP's you have imported. What is the role the server (virtual guest) is playing - are some of them more impacted than others when you consider their role?
If you are new to operations manager, common mistakes that lead to the high cpu symptom include importing all of the management packs you can find directly from the catalog without a tuning pass to make sure that in your environment, the right level of monitoring
for you is found. Some management packs are script heavy (such as the DHCP and DNS managemnet pack) and require considerable tuning to get working well.
The issue is very unlikely the XML file refresh (this is normal) - and the XML file is not used to drive any specific operations. You are probably confusing multiple operations being performed with cause/effect imagery.Microsoft Corporation
July 14th, 2010 8:10pm
I did try and download the XML hotfix but it would not install, I am currently investigating the problem on a Windows Server 2008 R2 machine that is not currently in production and is being tested. The MSXML6.dll file on the server has the version 6.30.7600.16385
in the hotfix the version is older than that.
Free Windows Admin Tool Kit Click here and download it now
July 15th, 2010 10:30am
I appreciate that there a lot of other tasks that happen during a particular operation, but why are there so many repeated calls to the same registry key when reading in the management pack XML files, I have other apps that read XML files and they do
not access the same registry keys, it appears that only the healthservice accesses these keys during the read operation.
I have looked at other areas i.e. we have a management pack that checks the status of scheduled tasks and by default there are many default tasks in 2008 and the MP launches a vbscript to check the status of each one, 70 in total but this operation only
lasts for ~10-15 seconds. The only time the server goes to 100% and repeats over and over again is when the XML read is done.
How can I stop servers reading in the XML files that do not apply to them, the server I am looking at will be a web server, why is it reading in the Exchange and ISA management packs xml files which are over 10MB in size?
July 15th, 2010 10:39am
As a crude test I have used XML Notepad (v2.5.2798.17141) and this will load the Exchange Management Pack XML file in 00:00:01.4062500 whereas the HealthService.exe will take 00:00:30.4544999 to load the same file.
XML Notepad is reading the file in 8K chunks whereas HealthService.exe is reading 2K chunks, when XML Notepad reads the file it will only show successive FASTIO_READ operations when the HealthService.exe reads the file it will show one FASTIO_READ operation
followed by numerous RegQueryKey, RegQueryValue, RegCloseKey and RegOpenKey operations, what is the HealthService.exe doing when it is reading in the XML file that no other application is doing, this is increasing the time it takes to read the file by a large
amount which is in turn making the HealthService.exe take longer in completing its tasks, which in turn makes people blame the HealthService.exe as it shows a lot of 100% CPU time.
If I open the Exchange XML file in XML Notepad while the HealthService.exe is running its tasks, the opening time increases to 5-6 seconds, so when the HealthService.exe process is running the system will slow, this is what people here are complaining about
with the servers, when the HealthService.exe is running the system slows down, and because they can see the HealthService.exe take up regular amounts of 100% CPU time its easy for them to blame that.
I am all up for looking at other areas for the CPU time usage, my point is why is the HealthService.exe taking so long to read the XML files and what is it doing between every read operation.
Free Windows Admin Tool Kit Click here and download it now
July 15th, 2010 11:44am
First item to check is to make sure antivirus software has exclusions setup, as well as script scan being disabled.
http://blogs.technet.com/b/smsandmom/archive/2008/11/05/recommendations-for-antivirus-exclusions-in-mom-2005-and-opsmgr-2007.aspx
Second item would be to update OpsMgr and OS using the info below.
Please take a look at Kevin holman's hot fix list for OpsMgr SP1 and apply the hot fixes that are needed. also look at Common related Windows Operating System Hotfixes to make sure your clients have the required updates. Once OpsMgr and the os has been patched
up the problem should go away.
http://blogs.technet.com/b/kevinholman/archive/2009/01/27/which-hotfixes-should-i-apply.aspx?wa=wsignin1.0
if these two steps did not correct the problem, the third step would be to follow Kevin's info in this post:
http://social.technet.microsoft.com/Forums/en-US/operationsmanagergeneral/thread/8060e1d1-072f-48e9-a30a-49e1783dd5ac/
Hope this helpsScott Moss MVP (Operations Manager) | President - System Center Virtual Users Group | Vice President - Atlanta Southeast Management Users Group (ATL SMUG)
July 15th, 2010 5:56pm
Again, you're chasing snakes in the weeds. Your 100% CPU isn't related to how long it takes to process/load/active workflows in an XML file. Look a the other advice from Scott below - make sure you have recent patches and the most recent cumulative
update for the product. Then start tuning your MPs if the 100% CPU continues.Microsoft Corporation
Free Windows Admin Tool Kit Click here and download it now
July 15th, 2010 7:37pm
Thanks Scott for that information I will go through that and see if that helps with the problems, a lot of good information there.
Dan, not so helpful you are basically saying nothing to see here move along, did you write the HealthService.exe is that is why your getting defensive, maybe you should move over to Apple and work in the Antenna Department ;-)
Thank you both for your help.
July 16th, 2010 10:40am
Steve, Were you able to solve this issue on your Server 2008 R2 devices? If so, can you share?
Free Windows Admin Tool Kit Click here and download it now
November 17th, 2010 12:19am
Hi Steve, were you able to solve this problem? I am experiencing the exactly same issue, Windows 2008 R2, CPU in 100% with MonitoringHost.exe and, on Process Monitor, several tentatives to read HKLM\Software\Microsoft\Cryptography\Offload and beyond register
keys.
Thanks,
Gustavo
August 23rd, 2011 5:01pm