Scanner Issues

Detecting a scanner problem is quite easy because the affected scanners are listed on top of the scanner homepage in the WebUI. But finding the root cause for the failure can be more difficult. Here are some tipps how to troubleshoot scanner issues:

Open the WebUI and go to SCANNER.

Critical configurations are listed on top of the page. There are two buttons for each scan category (Toplogy, Performance, Event, HFTCS): The left one shows the scan status, the right one the persist status.

Click on a red icon to see the log file for the selected item.

Set severity to "Error" to limit the amount of information.

In most cases, the error can now be narrowed down to either a configuration issue or a product defect:

IP address or password incorrect?
insufficient user rights to collect the requested data?
wrong protocol type?
access issues due to firewall or other limitations?

Known Issues

Compute

VMware

IBM PowerVM

Error Message: HTTP-Error 500 "Unable to connect to Database"

2021-04-08T09:17:37,383 ERROR [topology-scan_Worker-1]: Error executing call to [/rest/api/uom/ManagedSystem/<system_id>/<object_type>], got status [500 INTERNAL_SERVER_ERROR] with message [
<entry xmlns="http://www.w3.org/2005/Atom" xmlns:ns2="http://a9.com/-/spec/opensearch/1.1/" xmlns:ns3="http://www.w3.org/1999/xhtml">
<id>5fc9dc1c-03be-4e80-b8fb-4f9ea9dc0b4a</id>
<title>HttpErrorResponse</title>
<published>2021-04-08T09:17:46.640+02:00</published>
<author>
<name>IBM Power Systems Management Console</name>
</author>
<content type="application/vnd.ibm.powervm.web+xml; type=HttpErrorResponse">
<HttpErrorResponse:HttpErrorResponse xmlns:HttpErrorResponse="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/" xmlns="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/" xmlns:ns2="http://www.w3.org/XML/1998/namespace/k2" schemaVersion="V1_0">
<Metadata>
<Atom/>
</Metadata>
<HTTPStatus kb="ROR" kxe="false">500</HTTPStatus>
<RequestURI kxe="false" kb="ROR">/rest/api/uom/ManagedSystem/3870ebf9-dd78-31e5-a91e-719c4f86178b/NetworkBridge</RequestURI>
<ReasonCode kb="ROR" kxe="false">Unknown internal error.</ReasonCode>
<Message kb="ROO" kxe="false">com.ibm.pmc.rest.provider.exceptions.RESTProviderException: Exception While getting SEA:: Unable to connect to Database. </Message>
<RequestBody kb="ROO" kxe="false"/>
<RequestHeaders kxe="false" kb="ROO">
{x-forwarded-server=hmc101.localdomain, x-forwarded-host=172.16.146.192:12443, X-Transaction-ID=XT10011179, host=172.16.146.192:12443, connection=Keep-Alive, x-api-session=9PLm73Uh1Zg84V-wivc1TrsRpNLdrV-VbWeDWSfBEheHxNiuwGEgl_0OWwcLpFlVDaydA2z37DWTvHLMo9McsEwN9X_at8GfE5_ayfeF9Qjzf2EGlzYxJW0G4BzEaznehGmjRR7GfsY3ktUmcte6LT_-JHq5tCqUzfx4nVz6E9wu4dbG2Bo-DhJnkH81b1HLqpjCZyPQlAHf1fqpFOFWr29_7xq-GN_J5tiE1zwSXvY=, x-forwarded-for=172.16.168.84, accept-encoding=gzip,deflate, accept=application/vnd.ibm.powervm.uom+xml;, user-agent=Apache-HttpClient/4.5.12 (Java/11)}
</RequestHeaders>
</HttpErrorResponse:HttpErrorResponse>
</content>
</entry>
] (PowerVmClient)

BVQ Version: 2021.H1.3 and above

Suggested Action:

The Postgres DB on VIO servers which the HMC queries to get various system information (like virtual networks, virtual storage, etc.) is broken or Postgres service (vio daemon) is not running. This causes HTTP-500 error messages on both, the HMC and BVQ scanner.

The issue can be fixed using the following commands on the VIOS:

ssh padmin@${vios}
$ oem_setup_env
# stopsrc -s vio_daemon
# /usr/sbin/slibclean
# rm -rf /home/ios/CM
# startsrc -s vio_daemon -a '-d 4'

Error Message: HTTP-Error 500 "Exception while getting SEA"

2021-05-18T19:33:27,290 ERROR [topology-scan_Worker-6]: Error executing call to [/rest/api/uom/ManagedSystem/b8f44367-98bd-377d-8227-7db6208f1c4c/NetworkBridge], got status [500 INTERNAL_SERVER_ERROR] with message [
<entry xmlns="http://www.w3.org/2005/Atom" xmlns:ns2="http://a9.com/-/spec/opensearch/1.1/" xmlns:ns3="http://www.w3.org/1999/xhtml">
<id>7af9ea16-352a-4ab1-890f-1e24405102e7</id>
<title>HttpErrorResponse</title>
<published>2021-05-18T19:33:27.285+02:00</published>
<author>
<name>IBM Power Systems Management Console</name>
</author>
<content type="application/vnd.ibm.powervm.web+xml; type=HttpErrorResponse">
<HttpErrorResponse:HttpErrorResponse xmlns:HttpErrorResponse="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/" xmlns="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/" xmlns:ns2="http://www.w3.org/XML/1998/namespace/k2" schemaVersion="V1_0">
<Metadata>
<Atom/>
</Metadata>
<HTTPStatus kxe="false" kb="ROR">500</HTTPStatus>
<RequestURI kb="ROR" kxe="false">/rest/api/uom/ManagedSystem/b8f44367-98bd-377d-8227-7db6208f1c4c/NetworkBridge</RequestURI>
<ReasonCode kb="ROR" kxe="false">Unknown internal error.</ReasonCode>
<Message kb="ROO" kxe="false">com.ibm.pmc.rest.provider.exceptions.RESTProviderException: Exception While getting SEA:: The system is currently too busy to complete the specified request. Please retry the operation at a later time. If the operation continues to fail, check the error log to see if the filesystem is full. </Message>
<RequestBody kb="ROO" kxe="false"/>
<RequestHeaders kxe="false" kb="ROO">{x-forwarded-server=hmc3.labwi.sva.de, x-forwarded-host=hmc3.labwi.sva.de:12443, X-Transaction-ID=XT11438532, host=hmc3.labwi.sva.de:12443, connection=Keep-Alive, x-api-session=xnqKwOuPE9APo0rubodXReDeoXN2SnlAXIeEzEu4guge1pd6sg4oCF0WlAE94qpB7NjiX5q8L5xHLJMUuS4LWqRvxopaTucnrOqa6TACGCWhAMYJ4DekkrJtxlpM_s0GkNkoerZ5JSvutYojiYro9N2TNortma44FydeyORKQF260PAUjI2SLytd10mS8PJTpb9uzkbo6h0P0quXOSqRXg==, x-forwarded-for=10.10.120.73, accept-encoding=gzip,deflate, accept=application/vnd.ibm.powervm.uom+xml;, user-agent=Apache-HttpClient/4.5.12 (Java/11)}</RequestHeaders>
</HttpErrorResponse:HttpErrorResponse>
</content>
</entry>
] (PowerVmClient)

BVQ Version: 2021.H1.3 and above

Suggested Action:

Different problems can result in this error. One reason might be a full filesystem (as the error message itself suggests). Another reason might be a dodgy SEA adapter. Please run command

entstat -all entX

to see if there are Limbo Packets which indicate that the SEA has detected its physical network is not operational.

Network

Brocade (REST)

Error Message: "Max limit for REST sessions reached"

Got error from Brocade Switch with IP 10.10.101.147! (body: {
 "errors": {
  "error": [
   {
    "error-type": "application",
    "error-tag": "operation-failed",
    "error-app-tag": "Error",
    "error-path": "/rest/login",
    "error-message": "Max limit for REST sessions reached",
    "error-info": {
     "error-code": 14,
     "error-module": "auth"
    }

BVQ Version: 6.2 and above

Suggested Action:

By default, the SSH session limits on Brocade switches is set to 3. The number of SSH sessions can be increased up to 10 by using CLI command mgmtapp --config -maxrestsession <1...10>

Brocade (BNA)

Cisco

Storage

IBM SVC/Storwize/FlashSystem

Error Message: "rbash: line xxx: yyy Killed"

Catched Exception! (SSHConnectionImpl)
de.qualicision.bvq.exception.BvqScanException: SSH result contains "rbash: line xxx: yyy Killed"

BVQ Version: all

Suggested Action:

None.

This is an SVC limitation. The error typically disappears the next time the system is scanned.

Error Message: "the svc raised an error -> CMMVC6098"

Catched Exception! (SSHConnectionImpl)
de.qualicision.bvq.adapter.ssh.action.CommandErrorException: the svc raised an error -> CMMVC6098

BVQ Version: all

Suggested Action:

None.

The error occurs when SVC is busy , e.g. copying files between nodes. The error typically disappears the next time the system is scanned.

NetApp OnTap

EMC Unity

Troubleshooting

Scanner Issues

Known Issues

Compute

Storage