...

...

...

...

...

Introduction

To create To establish an access point for BVQ, you must create users on your various systems.
As an example: If you have 3 NetApp clusters, you have to create an user on every cluster,
the same workflow goes for every other system.
Using that specific new user on your system, you have to create a BVQ Scanner on your BVQ Server.
The BVQ Scanner will connect to the system, to collect data.

Procedure

The initial setup can be divided into 7 subsequent stages:

• Create users on your systems, with which BVQ is able to collect data
• Installation wizard which guides you through the setup
• Maintenance page where BVQ is checking the environment and its own files
• Initial Configuration helps you to create your first scanner and therefore enables BVQ to receive the first data
• Quick Tour explains the features of the BVQ webpage
• Grafana data sources must be adjusted after each implementation of a new HTTPS certificate
• BVQ GUI is an extensive tool, with which you are able to have direct access to your data

Prepare System
Brocade
Cisco

...

...

FOS REST API function calls are permitted or denied based on user privilege configurations determined by the role-based access control (RBAC) functionality in Fabric OS.
Implementing the following changes to your Brocade switches will not interrupt system operation or compromise performance.
Rest assured, these adjustments can be safely applied even on your running productive system.

Brocade REST API is supported on SAN switches running Fabric OS 8.2.1 and later but we recommend to use FOS 9 and higher. All switches running earlier versions cannot be monitored by BVQ! (see Supported Brocade systems).

Brocade SAN Switch User

For switches running FOS 9, the user needs at least the permissions of the default role basicswitchadmin.
For switches running FOS 8.2.x, the user needs at least the permissions of the default role admin.
For all versions, there is no support for default switch role user, because it has no permission to observe the RBAC class configure, that BVQ needs to gather information about the switch configuration.

Depending on the specific FOS level, the following user has to be added on each and every switch you want to scan:

FOS 9.x:
userconfig --add bvq -r basicswitchadmin -c basicswitchadmin -l 1-128 -h 128 -d "BVQ Scanner User" -p ChangeMeP@ssw0rd
passwd bvq
> <final_password>

FOS 8.2.x:
userconfig --add bvq -r admin -c admin -l 1-128 -h 128 -d "BVQ Scanner User" -p ChangeMeP@ssw0rd
passwd bvq 
> <final_password>

FOS 9.x:
userconfig --add bvq -r basicswitchadmin -d "BVQ Scanner User" -p ChangeMeP@ssw0rd
passwd bvq
> <final_password>

FOS 8.2.x:
userconfig --add bvq -r admin -d "BVQ Scanner User" -p ChangeMeP@ssw0rd 
passwd bvq 
> <final_password>

Increase Rest sessions

It is also an essential requirement to increase the number of allowed REST sessions to 10 on each switch in the fabric. This adjustment is a necessary step to ensure efficient operation.

mgmtapp --config -maxrestsession 10

mgmtapp --show
REST Configuration:
Interface State: Enabled
Effective Protocol: HTTPS only
HTTP State: Enabled
Session Count: 10
Throttling Configurations:
Sample Requests : 120
Sample Time (in sec) : 30
Idle Time (in sec) : 3

Gather information for BVQ Scanner configuration

Unlike BVQ versions prior to 2023.H1 where Brocade scanner configurations had to be configured for each virtual fabric, now only one scanner configuration per SAN is required. All switches that belong to the same fabric or are reachable by any virtual fabric on one of those switches will be detected and configured in a single scanner configuration. The switch IP used to discover the SAN must be one that has virtual fabrics feature in enabled - unless none of the switches in the fabric support this feature or have it enabled.

Switches in access gateway mode are not part of the fabric, and hence, have to be added to the configuration manually.

BVQ scanner configurations need the following input:

• Switch IP address or DNS name of one switch in the SAN. This does not need to be the principal but must be one that has virtual fabrics enabled if any of the switches in the SAN are using this feature.
• Protocol - http or https
• Switch username and password
• Port number (if not default)

• SSL / HTTPS certificate handling

  Code Block
  language bash theme DJango title Add BVQ user via FOS CLI (switches without VF)
  seccertmgmt show -cert https

  If you want to enable the BVQ Scanner Switch Check SSL certificate, you need to install a non-self-signed certificate on the switch.

Note:

If switches or virtual fabrics are added to or removed from the SAN, the scanner configuration needs to be adjusted manually. Edit the scanner configuration and select "Discover switches" to rediscover the SAN.

...

BVQ 2022.H2.1 and higher

BVQ collects topology and performance data from Cisco MDS switches using the Cisco MDS NX-API. This feature needs to be enabled on all Cisco MDS switches that should be monitored by BVQ.

switch# show feature
Feature Name         Instance State
-------------------- -------- -----
...
nxapi                1        disabled
...

switch# conf t
switch(config)# feature nxapi

switch# show feature
Feature Name         Instance State
-------------------- -------- -----
...
nxapi                1        enabled
...

Cisco SAN Switch User

The following user has to be added on each and every switch you want to scan:

cisco-BVQ-1# config terminal
Enter configuration commands, one per line.  End with CNTL/Z.
cisco-BVQ-1(config)# role name bvq-role
cisco-BVQ-1(config-role)# description Monitoring role for BVQ
cisco-BVQ-1(config-role)# rule 1 permit show
cisco-BVQ-1(config-role)# exit
cisco-BVQ-1(config)# role show
cisco-BVQ-1(config)# show role
Role: bvq-role
  Description: Monitoring role for BVQ
  Vsan policy: permit (default)
  -------------------------------------------------
  Rule    Type    Command-type    Feature
  -------------------------------------------------
  1       permit  show            *

cisco-BVQ-1(config)# username bvq-user password P@ssw0rd role bvq-role
cisco-BVQ-1(config)# exit
cisco-BVQ-1# show user-account
user:bvq-user
        this user account has no expiry date
        roles:bvq-role

config terminal
role name bvq-role
description Monitoring role for BVQ
rule 1 permit show
exit
username bvq-user password P@ssw0rd role bvq-role
exit

...

...

System preparation steps

For the communication of the BVQ SVC Scanner with the SVC CLI a user account on each SVC is mandatory, which should be at least member of the group 'Monitor'.

...

Create a BVQ user on the system

...

With SVC CLI:

svctask mkuser -name bvq -usergrp Monitor -password P@ssw0rd

For code level lower than 8.4 use:

svctask mkuser -name bvq -usergrp Administrator -password P@ssw0rd

...

Check NTP, time and time zone

...

Please check if an NTP server is configured: we strongly recommend to use an NTP server to synchronize the time of all systems (SVC Nodes, Windows):

svcinfo lssystem | grep 'ntp_IP_address'

Set your NTP Server: Synchronize your SVC cluster with a specific NTP server:

svctask chcluster -ntpip <IP address of NTP Server>

Adjust SVC cluster time zone: Set the time zone of your SVC cluster with:

svctask settimezone -timezone 360

Show SVC clock settings: Check the current time setting on your SVC:

svqueryclock

...

Check performance statistics interval

...

The SVC performance statistics are generated regularly (in intervals) by the SVC and picked up by the BVQ SVC Scanner. BVQ supports all intervals possible in the SVC (1min to 60min). We recommend to set the time interval to 1 minute.

Check statistics status and frequency: Use this command to see if and how often your system collects statistics:

svcinfo lssystem | while read key value; do [[ "$key" =~ ^(statistics_status|statistics_frequency)$ ]]&& echo "$key $value"; done

Enable statistics collection: Activate collection of system statistics:

statistics_status on

Set statistics collection frequency: Define how frequently the system collects statistics:

statistics_frequency 1

Start statistics collection: Begin collecting system statistics at your defined frequency:

svctask startstats -interval 1

Clear I/O statistics dumps: Clean old I/O statistics data from all nodes in the SVC with this command:

svcinfo lsnode -nohdr | while read id rest ; do svctask cleardumps -prefix /dumps/iostats $id ; done

...

A user is required for the operation of the BVQ Scanner, which should at least have readonly Role permissions on the ONTAP Cluster.

Please create this user before the configuration of the BVQ Scanner.

We recommend the name: bvq

sec login create -user-or-group-name bvq -application http -authentication-method password -role readonly

sec login create -user-or-group-name bvq -application ontapi -authentication-method password -role readonly

...

...

...

A user is required for the operation of the BVQ PowerVM Scanner who should at least have read-only (hmcviewer) access to the HMC.

Please create this user before configuring the BVQ Scanner. We recommend to name the user "bvq"

Please open the user properties dialogue and select "Allow remote access via the web"

...

...

Enable performance data collection

BVQ can only collect performance statistics if "Data Collection" on the managed systems and LPARs is enabled.

• For a better performance of the HMC, we recommend to change the Performance Data Storage value to "1".

...

BVQ Scanner configuration

To configure a PowerVM scanner in BVQ the following information is required:

• IP address or hostname of the HMC
• User and password of the HMC user for BVQ

Starting with BVQ 2023.H1: Redundant HMCs managing the same systems must be configured in the same PowerVM scanner. Otherwise, the managed systems will appear twice in BVQ. Define the most powerful HMC first, because the order of HMCs determines the order in which they are scanned by BVQ. Additional HMCs managing other systems should be configured in an additional PowerVM scanner.

Up to BVQ 2022.H2: Typically, two redundant HMCs manage the same IBM Power systems. Please ensure that only one scanner is created for one of the HMCs to avoid duplication in BVQ.

...

AIX and Linux are the first BVQ platforms where data is not pulled from the systems by the BVQ scanner. Instead, data is sent (pushed) from the OS on the LPARs to the BVQ Server by an BVQ OS Agent using SCP. This means, an ssh-server on the BVQ Server is receiving data from the OS instances. Once an AIX or Linux BVQ Scanner is configured, the ssh-server is being started and listening on port 2222.

BVQ Scanner configuration

To configure an AIX or Linux BVQ scanner the following information is required:

• NAME - Name of the AIX or Linux scanner
• INSTANCE GROUP NAME - Select a name which is used to group all AIX or Linux Instances (=partitions) together that are running the BVQ OS Agent for AIX or Linux
• USERNAME - This user authorizes the SCP transfer from the AIX or Linux Instances to the BVQ Server. It will be configured during the installation process

OS Agent installation

The BVQ Agent for AIX or Linux RPM installation package is automatically generated once a new BVQ AIX or Linux scanner configuration is being created in the BVQ WebUI. After the "Save"-button is pressed, the RPM package is automatically generated and can be downloaded directly. Further installation instructions can be found in the scanner configuration page or the readme included in the RPM download package.

OS User requirements

...

Alternatively, the BVQ AIX agent can be rolled out automatically to many systems using an AIX NIM server. The download package for AIX includes a script that helps configuring the NIM server.

...

A user is required for the operation of the BVQ VMware Scanner, who should at least have read-only access to the VMware vCenter system. The read-only permission for the user must be defined at the vCenter level. Permissions in a lower level (e.g. Datacenter, Cluster, ...) will lead to scan errors.

Please create this user before configuring the BVQ Scanner. We recommend to name the user: bvq

Create or select the right user role

• Go to user roles
  

  Expand
  title screenshot ...
  Info icon false    Image Removed

• Duplicate (1) the read-only role and store it as BVQ-Read-only (2) and add the following privileges (3)
  Datastore - browse datastore
  Profile driven storage - view
  Storage views - view

  Expand
  title screenshot ...
  Info icon false Image Removed

Create the BVQ User for the vCenter

• Create the bvq user with the role "BVQ-read-only"
  create it as vsphere.local or as AD user - please remember to add it correctly into the scanner dialog later

  Expand
  title screenshot ...
  Info icon false Image Removed

• Add the user to the vCenter
  Add the  user to the vCenter (4) and do not forget to define it for all children

  Expand
  title screenshot
  Info icon false Image Removed

Add the right vCenter Statistics

• Interval duration has to be 5 minutes

• Level 2 is sufficient for standard VMware
  Level 3 should be used for VSANs

  Expand
  title screenshot
  Info icon false Image Removed

Gather information for BVQ Scanner configuration

BVQ scanners need the following information to be configured for each vCenter system:

• vCenter IP address or hostname
• vCenter user domain
• vCenter ID and password of the bvq user

Preparation for the BVQ Server

For BVQ Servers which are gathering information from NetApps and vCenters, the correct DNS configuration is important.
Make sure that the BVQ Server, NetApp systems and vCenters are in the same domain and have the same DNS server configured. 

This is required to match the DNS-Name of the NFS Datastores to the corresponding IP Adresses of the NFS file shares on NetApp systems.

...

Kubernetes (k8s) clusters are scanned via 2 different methods:

• Kubernetes API Server for topology information
• BVQ Prometheus Server for topology & performance information

Kubernetes API Server

To gain access to the k8s API server the following preparations must be made:

1. Create a Prepare System (CRD) to set up a k8s cluster as master grouping object (MGO) definition for BVQ
2. Create a Prepare System instance (binded to the CRD) for the k8s cluster
3. Create a Prepare System to get read-only (get, list, watch) access to the k8s cluster
4. Create a Prepare System for authentication

5. Create a Prepare System to bind the ServiceAccount to the ClusterRole

 Use kubectl apply -f  to create the expected objects. You can edit & use the all in one preparation YAML file to set up all requirements in one step.
(make sure all 5 objects are created properly - sometimes Prepare System creation fails due to the delayed creation of the Prepare System)

CustomResourceDefinition

Create a CustomResourceDefinition (CRD) to set up a k8s cluster as master grouping object (MGO) definition for BVQ

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mastergroupingobjects.bvq.sva
spec:
  group: bvq.sva
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                clusterName:
                  type: string
                  description: Cluster-Name
                customer:
                  type: string
                  description: Customer-Name
                location:
                  type: string
                  description: Location where the Cluster is located at
                dc:
                  type: string
                  description: Datacenter-Name
                contact:
                  type: string
                  description: Customer-Contact-Name
                email:
                  type: string
                  description: E-Mail-Address of the Contact
                phone:
                  type: string
                  description: Phone-Number of the Contact
  scope: Cluster
  names:
    plural: mastergroupingobjects
    singular: mastergroupingobject
    kind: MasterGroupingObject
    shortNames:
    - mgo

MasterGroupingObject

Create a MasterGroupingObject instance (binded to the CRD) for the k8s cluster

 Edit/adjust the values for clusterName, customer, location, dc, contact, email  & phone  to the required information

 IMPORTANT: clusterName  will represent the name of the k8s cluster within BVQ, so choose a meaningful name (example would be: Prod-Cluster-01)

apiVersion: bvq.sva/v1
kind: MasterGroupingObject
metadata:
  name: bvq-mgo-k8s
  labels:
    bvq: mgo
spec:
  clusterName: Prod-Cluster-01
  customer: Customer Inc.
  location: Berlin, Germany
  dc: Example-DC-01
  contact: Max Mustermann
  email: max.mustermann@customer.de
  phone: +49-171-1234-56789

ClusterRole

Create a ClusterRole to get read-only (get, list, watch) access to the k8s cluster

 Read only permissions (get, list, watch) are required
apiGroups may be applied via a wildcard ('*') to get access to all api groups, otherwise apiGroups given in the example must be set

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: bvq-scanner-rl
rules:
  - verbs:
      - get
      - watch
      - list
    apiGroups:
      - ''
      - apiextensions.k8s.io
      - apps
      - batch
      - bvq.sva
      - networking.k8s.io
      - storage.k8s.io
      - discovery.k8s.io
      - scheduling.k8s.io
    resources:
      - '*'

ServiceAccount

Create a ServiceAccount for authentication

 The Token created for this ServiceAccount is needed to set up a BVQ scanner config for the k8s cluster
namespace may be adjusted to another kubernetes namespace. Remember to edit the namspace set in the Prepare System

 IMPORTANT: With k8s version 1.24 the LegacyServiceAccountTokenNoAutoGeneration feature gate is beta, and enabled by default (see here). Use this guide to create a non-expiring token (recommended)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: bvqscan
  namespace: default

ClusterRoleBinding

Create a ClusterRoleBinding to bind the Prepare System to the Prepare System

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: bvq-scanner-sa-bnd
subjects:
- kind: ServiceAccount
  name: bvqscan
  namespace: default
roleRef:
  kind: ClusterRole
  name: bvq-scanner-rl
  apiGroup: rbac.authorization.k8s.io

BVQ Prometheus Server

To get performance and topology data a custom bvq-prometheus stack must be deployed in the k8s cluster via helm. This helm chart will install a bvq-prometheus server as a deployment with a 8GB persistent volume (configurable via values.yaml) and bvq-prometheus-node-erxprters as a DaemonSet (helm dependency).

See values.yaml and other configuration files in the bvq-prometheus-helm.zip file for further information about the bvq-prometheus configuration.

Execute the following steps to deploy the bvq-prometheus helm chart to the k8s cluster:

• Create a namespace (e.g. bvq-prometheus) for the prometheus stack:
  kubectl create namespace bvq-prometheus 
• Unzip helm files → bvq-prometheus-helm.zip
• For external communication an ingress for the bvq-prometheus server is needed. Edit prometheus.ingress.hosts  in values.yaml to set a proper ingress.
• Run helm dependency build / helm dependency update 

• Install the helm chart via helm install -n bvq-prometheus -f values.yaml bvq-prometheus ./ 

  Code Block
  language bash title helm install -n bvq-prometheus -f values.yaml bvq-prometheus ./ collapse true
  ▶ helm install -n bvq-prometheus -f values.yaml bvq-prometheus ./ NAME: bvq-prometheus LAST DEPLOYED: Thu Dec 15 11:00:08 2022 NAMESPACE: bvq-prometheus STATUS: deployed REVISION: 1 TEST SUITE: None

• Check the installation with kubectl get pods -n bvq-prometheus  - A pod called bvq-prometheus-* and a set of bvq-prometheus-bvq-node-exporter-* pods should be in running state
  

  Code Block
  language bash title kubectl get pods -n bvq-prometheus collapse true
  ▶ kubectl get pods -n bvq-prometheus NAME READY STATUS RESTARTS AGE bvq-prometheus-5b8cd79d79-r587m 1/1 Running 0 64s bvq-prometheus-bvq-node-exporter-jz46z 1/1 Running 0 2s

Gather information for BVQ Scanner configuration

BVQ scanners need the following information to be configured for each k8s cluster:

• API server IP address or DNS name (FQDN) - Default TCP port: 6443
• API Token of the bvqscan Prepare System
• Prometheus URL or IP (if NodePort service is used)
• Prometheus user & password (optional, if BasicAuth of Prometheus is used)

Preparation for the BVQ Server

...

it's necessary to create user accounts on the respective systems you wish to monitor.

For instance, if you have three NetApp clusters, you'll need to create a user account on each cluster.

This procedure is applicable to all systems you aim to connect to BVQ. 

After creating a specific new user on a system, proceed to set up a BVQ Scanner on your BVQ Server using that user account.

The BVQ Scanner will then establish a connection to the system to collect the requisite data.