Kubernetes (k8s) clusters are scanned via 2 different methods: Kubernetes API ServerTo gain access to the k8s API server the following preparations must be made: - Create a CustomResourceDefinition (CRD) to set up a k8s cluster as master grouping object (MGO) definition for BVQ
- Create a MasterGroupingObject instance (binded to the CRD) for the k8s cluster
- Create a ClusterRole to get read-only (get, list, watch) access to the k8s cluster
- Create a ServiceAccount for authentication
Create a ClusterRoleBinding to bind the ServiceAccount to the ClusterRole
Use kubectl apply -f to create the expected objects. You can edit & use the all in one preparation YAML file to set up all requirements in one step. (make sure all 5 objects are created properly - sometimes MasterGroupingObject creation fails due to the delayed creation of the CustomResourceDefinition) CustomResourceDefinitionCreate a CustomResourceDefinition (CRD) to set up a k8s cluster as master grouping object (MGO) definition for BVQ Code Block |
---|
language | yml |
---|
title | mgo-crd.yaml |
---|
linenumbers | true |
---|
collapse | true |
---|
| apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: mastergroupingobjects.bvq.sva
spec:
group: bvq.sva
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
clusterName:
type: string
description: Cluster-Name
customer:
type: string
description: Customer-Name
location:
type: string
description: Location where the Cluster is located at
dc:
type: string
description: Datacenter-Name
contact:
type: string
description: Customer-Contact-Name
email:
type: string
description: E-Mail-Address of the Contact
phone:
type: string
description: Phone-Number of the Contact
scope: Cluster
names:
plural: mastergroupingobjects
singular: mastergroupingobject
kind: MasterGroupingObject
shortNames:
- mgo |
MasterGroupingObjectCreate a MasterGroupingObject instance (binded to the CRD) for the k8s cluster Edit/adjust the values for clusterName, customer, location, dc, contact, email & phone to the required information IMPORTANT: clusterName will represent the name of the k8s cluster within BVQ, so choose a meaningful name (example would be: Prod-Cluster-01) Code Block |
---|
language | yml |
---|
title | mgo-instance.yaml |
---|
linenumbers | true |
---|
collapse | true |
---|
| apiVersion: bvq.sva/v1
kind: MasterGroupingObject
metadata:
name: bvq-mgo-k8s
labels:
bvq: mgo
spec:
clusterName: Prod-Cluster-01
customer: Customer Inc.
location: Berlin, Germany
dc: Example-DC-01
contact: Max Mustermann
email: max.mustermann@customer.de
phone: +49-171-1234-56789 |
ClusterRoleCreate a ClusterRole to get read-only (get, list, watch) access to the k8s cluster Read only permissions (get, list, watch) are required apiGroups may be applied via a wildcard ('*') to get access to all api groups, otherwise apiGroups given in the example must be set Code Block |
---|
language | yml |
---|
title | cluster-role-bvqscan.yaml |
---|
linenumbers | true |
---|
collapse | true |
---|
| apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: bvq-scanner-rl
rules:
- verbs:
- get
- watch
- list
apiGroups:
- ''
- apiextensions.k8s.io
- apps
- batch
- bvq.sva
- networking.k8s.io
- storage.k8s.io
- discovery.k8s.io
- scheduling.k8s.io
resources:
- '*' |
ServiceAccountCreate a ServiceAccount for authentication The Token created for this ServiceAccount is needed to set up a BVQ scanner config for the k8s cluster namespace may be adjusted to another kubernetes namespace. Remember to edit the namspace set in the ClusterRoleBinding IMPORTANT: With k8s version 1.24 the LegacyServiceAccountTokenNoAutoGeneration feature gate is beta, and enabled by default (see here). Use this guide to create a non-expiring token (recommended) Code Block |
---|
language | yml |
---|
title | bvq-serviceaccount.yaml |
---|
linenumbers | true |
---|
collapse | true |
---|
| apiVersion: v1
kind: ServiceAccount
metadata:
name: bvqscan
namespace: default |
ClusterRoleBindingCreate a ClusterRoleBinding to bind the ServiceAccount to the ClusterRole Code Block |
---|
language | yml |
---|
title | cluster-role-binding-bvqscan-sa.yaml |
---|
linenumbers | true |
---|
collapse | true |
---|
| apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: bvq-scanner-sa-bnd
subjects:
- kind: ServiceAccount
name: bvqscan
namespace: default
roleRef:
kind: ClusterRole
name: bvq-scanner-rl
apiGroup: rbac.authorization.k8s.io |
BVQ Prometheus ServerTo get performance and topology data a custom bvq-prometheus stack must be deployed in the k8s cluster via helm. This helm chart will install a bvq-prometheus server as a deployment with a 8GB persistent volume (configurable via values.yaml ) and bvq-prometheus-node-erxprters as a DaemonSet (helm dependency). See values.yaml and other configuration files in the bvq-prometheus-helm.zip file for further information about the bvq-prometheus configuration. Execute the following steps to deploy the bvq-prometheus helm chart to the k8s cluster: - Create a namespace (e.g. bvq-prometheus) for the prometheus stack:
kubectl create namespace bvq-prometheus - Unzip helm files → bvq-prometheus-helm.zip
- For external communication an ingress for the bvq-prometheus server is needed. Edit
prometheus.ingress.hosts in values.yaml to set a proper ingress. - Run
helm dependency build / helm dependency update Install the helm chart via helm install -n bvq-prometheus -f values.yaml bvq-prometheus ./ Code Block |
---|
language | bash |
---|
title | helm install -n bvq-prometheus -f values.yaml bvq-prometheus ./ |
---|
collapse | true |
---|
| ▶ helm install -n bvq-prometheus -f values.yaml bvq-prometheus ./
NAME: bvq-prometheus
LAST DEPLOYED: Thu Dec 15 11:00:08 2022
NAMESPACE: bvq-prometheus
STATUS: deployed
REVISION: 1
TEST SUITE: None |
Check the installation with kubectl get pods -n bvq-prometheus - A pod called bvq-prometheus-* and a set of bvq-prometheus-bvq-node-exporter-* pods should be in running state
Code Block |
---|
language | bash |
---|
title | kubectl get pods -n bvq-prometheus |
---|
collapse | true |
---|
| ▶ kubectl get pods -n bvq-prometheus
NAME READY STATUS RESTARTS AGE
bvq-prometheus-5b8cd79d79-r587m 1/1 Running 0 64s
bvq-prometheus-bvq-node-exporter-jz46z 1/1 Running 0 2s |
BVQ scanners need the following information to be configured for each k8s cluster: - API server IP address or DNS name (FQDN) - Default TCP port: 6443
- API Token of the bvqscan ServiceAccount
- Prometheus URL or IP (if NodePort service is used)
- Prometheus user & password (optional, if BasicAuth of Prometheus is used)
Preparation for the BVQ ServerFor BVQ Servers which are gathering information from Kubernetes clusters, the correct DNS configuration is important. Make sure that the BVQ Server & Kubernetes clusters are in the same domain and have the same DNS server configured.
|