WMI not Working Few weeks After Server reboot

Recently I had this issue. On few physical Servers I was getting WMI errors.

Issues were occurring on physical HP servers with WBEM Providers installed.

These were the symptoms:

1. WMI Service was running
2. When trying to test simple WMI Query I got an error
3. When trying to Restart WMI Service the service hangs in Stopping state
4. After Server reboot WMI started to work as expected, but only for a week or so..

Soon I started to think that this is a WMI memory leak. I have checked % Memory used history and indeed graph looked something like this:

MemoryLeak

I did not past the scale but please note that this is gradual increase (Scale in days, but certainly not in minutes)

Then I checked wmiprvse.exe Since It was not using a lot of memory I checked svchost.exe

Indeed svchost was using large amount of Memory; approximately 7 GB.

I found this great post: “WMI: High Memory Usage by WMI Service or Wmiprvse.exe”

Then I checked if wmiprvse was running under this svchost process. There are multiple methods on how to achieve this. You can select svchost in task manager, right click it and Select: “Go to Service(s)”. You could use, process explorer..

Another method we can find in Above post is to Run in CMD: sc queryex winmgmt

Scqueryex

Then I said OK before we go into some serious debugging lets do first things first. I remembered that Microsoft has released some patches which also addressed WMI leak issues. So I patched the server.

That did not help. So the next step was to Update HP Software and Drivers with HP SUM.

I downloaded: HP Service Pack for ProLiant (SPP) Version 2014.09.0 (the page was not working by the way when I was writting this post so I did not add a link).

The update procedure I Used was to:
1. Copy “Swpackages” folder from downloaded ISO to the server that will be patched.
2. Open CMD in elevated mode. Browse to swpackages folder and run: hpsum.bat
3. Run baseline library
4. Once it completes run:  Local host guided update
5. Selected only these two:
HPhomepageUpgrade
6. After server reboot I selected only:
WBEMupdate
7. Rebooted once again

And…

After a few day the Memory consumption looked like this:

MemoryLeakBeforeAfter

That is it. It looks like Memory leak was caused by HP Insight Management WBEM Providers.

Few other notes:

For WMI test I wrote this simple script (it is a part of my Custom OS SCOM management pack):
Write-Host “Testing WMI..`n——————————————–”
$WMIServiceStatus=(get-service Winmgmt).Status
Write-Host “WMI Service is: $WMIServiceStatus`n——————————————–”

$TestWMIConnection=(Get-WmiObject -Class win32_OperatingSystem | fl CSName,Caption,FreePhysicalMemory,OSArchitecture)
$GetWMISettings=(Get-WmiObject -Namespace “root\cimv2″ -Class Win32_WMISetting | Format-List *)

$StringLines=”——————————————–”
$StringConnect=”Connecting to WMI..`nSystem name, OS Version, Memory and Architecture should be displayed below: ”

$StringSett=”Displaying WMI Settings… ”
$StringConnect
$TestWMIConnection
$StringLines

$StringSett
$GetWMISettings
$StringLines

To Verify/Salvage WMI Repository:

Verify:
Open CMD in elevated mode and Run: winmgmt /verifyrepository

If inconsistencies found you can Salvage repository by:
1) Stopping WMI service
2) Rename directory: C:\windows\system32\wbem\repository
3) Start WMI service
4) Rebuild WMI rebuild repository by Running in CMD elevated mode: winmgmt /salvagerepository

C U Soon

Jure

Advertisements
This entry was posted in HP, Troubleshooting, Windows Server and tagged , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s