SymptomsThe service account used by HP Systems Insight Manager gets frequently locked out after failed authentication due to incorrect username or password.
The following events are present in the Application event log in Windows: Event Type: Error
Event Source: HP Systems Insight Manager
Event Category: None
Event ID: 3
Date: 24/03/2009
Time: 23:15:25
User: N/A
Computer: CAMHPSIM01
Description:
mxdomainmgr(error): Received an remote exception doing lookup to RMI registry. ----- Event Type: Error
Event Source: HP Systems Insight Manager
Event Category: None
Event ID: 3
Date: 24/03/2009
Time: 23:18:08
User: N/A
Computer: CAMHPSIM01
Description:
com.hp.mx.database.DbVerificationException: Error loading database verification handler 'com.hp.mx.database.MsSqlVerificationHandler'
com.hp.mx.database.DbVerificationException: Error accessing database ----- Event Type: Warning
Event Source: HP Systems Insight Manager
Event Category: None
Event ID: 2
Date: 25/03/2009
Time: 00:12:57
User: N/A
Computer: CAMHPSIM01
Description:
Restarting MxDomainManager The following event is continuously repeated in the application log of HP SIM (mxdomainmgt.x.log): [date] [time],581 INFO [HPSIM_DEBUG] [Panic Logger-0] (Identification Via Discovery:1) BPX during credential decryption Given final block not properly padded CauseThis can happen when the authentication information saved in HP SIM's database is wrong and the same user account is used to run HP SIM, connect to the database and connect to the monitored clients. When authentication information saved in the database becomes invalid and HP SIM tries to connect to the client machines using that data, many authentication failures can cause the service account to be locked (depending on the account policies enforced). More specifically, this has been found to be caused by the WBEM credentials but the same problem could be generated by sign-in or WS-Man credentials. ResolutionTo quickly find out the systems against which HP SIM is trying to authenticate using the specific account, from a command propmt run the following command: mxnodesecurity -l Output from this command has fixed width and can easily be imported in a spreadsheet to be filtered. You will be looking for something similar to the following: NODENAME PROTOCOL USERNAME PASSWORD TRYOTHERS
simserver wbem domain\user ******** Yes There are 2 ways to solve the problem. Option 1 is recommended when only a few credentials need to be amended; option 2 should be avoided if possible, but it's the only (known) way to do this in bulk. OPTION 1Open HP SIM
For each one of the systems having the wrong credentials, do the following: - In the "All Systems" view, select the system
- From the "Options" menu, select "Security", "Credentials", "System Credentials"
- From the table, select the wrong credentials and click on "Edit system credentials..."
- In the "Edit System Credentials" panel, click on "Show advanced protocol credentials"
- Click on the tab(s) related to the wrong credentials, amend them and click on "OK" OPTION 2The following procedure should be executed using SQL Server Management Studio to connect to the server hosting HP SIM's database. The database used to host HP SIM's data is assumed to be Microsoft SQL Server 2005. Use the following procedure: - Open SQL Server Management Studio and connect to the server hosting SIM's database
- In the left pane, expand "Databases", "Insight_v50_0_xxxx" (where "Insight_v50_0_xxxx" is HP SIM's database), "Tables"
- Right-click "dbo.NodeCredentialMap" and select "Script Table as", "DELETE To", "New Query Editor Window"
- The query will be similar to the following: DELETE FROM [Insight_v50_0_14203480].[dbo].[NodeCredentialMap] WHERE protocol = 'wbem'
(change "wbem" with the relevant protocol if needed)
- Click on "Execute" and check the result
- Running mxnodesecurity -l on the CMS should produce no results for the wbem protocol (or the alternative one used in the above query) StatusThis problem has been noticed after reinstalling HP Systems Insight Manager v5.3 and there is no proof that it could happen in normal installation/upgrade scenarios. Applies toHP Systems Insight Manager 5.3 Related links
Author's commentWe had a problem upgrading SIM from v5.2 SP2 to v5.3 that forced us to un-install the latter (the upgrade was successful but the application was failing continuously) and re-install it. After reinstallation, instead of picking up the old DB, SIM created a new one, so we had to point the new installation to the old database manually, using part of HP's procedure to move the DB to a different server. This made the database available to SIM but the saved credentials didn't work anymore. Every time the "Hardware Status Polling for Servers" task was executed, the service account was locked out, connection to the database was lost and SIM crashed. The use of different user accounts would have probably mitigated the problem, in the end we had to use option 2 of this procedure to clean the database. I hope this will help saving the 2 days that I spent to find a solution to this problem.
|
HP 
