Introduction:

Alerting and the rule set within it, is an important building block for monitoring a complex environment using thresholds.
These rules contribute to how precise a check and the monitoring of a specific system takes place.
It is also of great importance, in which timeframe a rule violation takes place and how often.
An information about each rule violation can be useful, it also can lead to an unwanted flood of notifications, which are rather disturbing.

With BVQ 2022.H1 the Occurrence counter is available as a new alerting functions.
This introduced a very useful tool, with which you are able to determine and create thresholds regarding your needs.

In the following, we will explain the use and benefits of this innovation.

Alerting - Occurrence counter:

The Occurrence counter is part of the Alerting system in BVQ.
Until BVQ release 2022H1.4, any change in the status of a rule has been directly reported with the respecting alert level, for that rule.

However, a single occurrence of a threshold break might not be a sufficient reason for an alert.
With the introduction of the Occurrence counter, the user has the ability to decide within a user-defined alert rule, when he wants to be informed about the status change of a rule.

For this purpose BVQ has introduced three new options with the new Occurrence counter:

1. Violations per timeframe (sliding window)

2. Violations per SLA interval (fixed window)

3. Consecutive violations (useable in both, sliding or fixed window)

The user has the option to keep the default direct trigger mode active, which means the Occurence counter stays disabled,
The user has also the option to activate the all new Occurrence counter and use these new options to specify in more detail when an alarm is triggered (delayed trigger mode).

Where can I find the Occurrence counter and how can I enable it:

By default, the Occurrence counter is disabled.
For enabling, you have to activate the switch and add one or more counter definitions to your BVQ alert condition.

Figure 1

Open

Figure 1: Use Occurrence counter

After enabling the Occurrence counter, you can decide on a method to choose from  → Consecutive violations or Violations per time

Both methods can also be used in the Service Level Agreement (SLA) mode.
The capabilities and the usage will also be explained in this document.

INFO: To use this on Predefined alert rules, you have to clone the rule and edit the clone.
The Occurrence counter can only be used within custom alert rules.

How to enable and use the different options:

As illustrated in Figure 1, after enabling this option you can decide which method to choose from.
Both methods differ in the way, a status change of the respective alert condition is reported.
In the alert condition is defined, what value the threshold has and in which time frame the break of a threshold is resulting in an alert.  

Consecutive violation determines the quantity of errors that occur directly one after the other.
Violations per timeframe determines the quantity of errors that occur per time interval. 

Please be aware that you can add multiple Occurrence counters to each condition.
It can be quite useful to define multiple alert levels with separate counts and to have the ability to combine "per time" with "consecutive" violations.
You can reorder their sequence by dragging them at the desired position.
The conditions are checked from top to bottom, if a matching condition is recognized, all the following conditions below will not be checked.

INFO: Please note that a mixture of SLA and non SLA mode within a single Alert Condition is not possible.

Figure 2

Consecutive violations	Open	Type Type of this Occurrence counter                                                      Consecutive violations - define how many consecutive violations in a row must occur before the condition is fulfilled Amount Number of violations Maximum number of violation occurrences that are counted until the Alert level is raised. Alert level BVQ Alert level Desired Alert level, one of: OK, INFO, WARN, ERROR, UNKNOWN
Violations per timeframe	Open	Type Type of this Occurrence counter                                                      Violations per timeframe - define how often in a certain timeframe a condition must match until the condition is fulfilled Time Minutes Size of the sliding timeframe, in minutes. Only available for type "Violations per time" if SLA mode is turned off. Amount Number of violations Maximum number of violation occurrences that may be counted until the Alert level is raised. Alert level BVQ Alert level Desired Alert level, one of: OK, INFO, WARN, ERROR, UNKNOWN
Violations per SLA interval (fixed window)	Open	Type Type of this Occurrence counter Violations per SLA interval - define how often in a fixed timeframe a condition must match until the condition is fulfilled Time Open If the SLA Mode is enabled the time setting is not available. The number of Alert conditions within the SLA interval will be counted. Amount Number of violations Maximum number of violation occurrences that may be counted until the Alert level is raised. Alert level BVQ Alert level Desired Alert level, one of: OK, INFO, WARN, ERROR, UNKNOWN

Figure 2: Differences between the options

Illustration of the different methods:

The following chart illustrates the different methods, using performance data like the latency and IO/s of an MDisk. 

RED:         Every violation of the given Alert rule will be informed. (Occurrence counter OFF) - (direct trigger mode)

BLUE:       Only a consecutive sequence of exceedances triggers an alarm. (Occurrence counter ON - Consecutive Violations) - (delayed trigger mode)

YELLOW:  Only a certain amount of exceedances within a specified timeframe triggers an alarm. (Occurrence counter ON - Violations per Time) - (delayed trigger mode)

Figure 3

Open

Figure 3: Real Life example and illustration of the different methods

Example & description

This table describes the different options using a real-life example.
More detailed information about each option is given, as well as a description of how to activate one of the options.

Figure 4

Methods of choice	Example	How does it work?	What to fill in

Methods of choice	Example	How does it work?	What to fill in
Default (direct trigger mode)	Open	Each exceeding of a predefined threshold triggers an event within the designated warning level. (Info, Warning, Error) If a latency exceeds 3ms, the threshold is reached and an error message is triggered.  The example with the red border shows only one of these events for clarification, but this behavior would apply to all further exceedances.   The default will raise an Error event as soon as the rule is violated the first time.	Open
Consecutive Violations	Open	Only when a set of 5 consecutive errors occur, an event is triggered within the designated warning level. (Info, Warning, Error) This condition is shown inside the blue frame and actually occurs only once in this performance chart.   The latency for the MDisk is 5 times above the 3ms threshold and therfore this rule will raise an Error.	Open
Violations per timeframe	Open	The yellow box shows the possibility of setting "Violations per time". Here, a number of violations per time is specified. As soon as this number is reached, the designated warning level is displayed. However, if this value is no longer exceeded, the status of the alert rotates back to the next better value.   In this example we have specified a number of 3 Violations per 60 min. timeframe. With the exceedance of this an alert message will be raised.	Open

Figure 4: The different options using a real-life example

When using the non-SLA - sliding window mode, the status of a rule varies, depending on the current value.
If in the next measurement the value falls below a critical value again, this will be reflected in the status of the alert rule and it will change from e.g. Error Level back to an OK state.
Relevant for this is the PI timing and the time specified in the Sliding window. 
Here, an additional possibility has been created to maintain this state in the long term, over a certain SLA time interval. (fixed window)

SLA timing mode for BVQ Occurrence counter

As stated above, both options of the occurrence counter can also be used in an SLA mode.
The SLA mode allows the setting of a so-called fixed time window. This fixed time window specifies a period of time, in which the warning rules are applicable.
So, if it has to be proved that a certain rule is not exceeded within a certain period of time, this option is the right setting. 

The main difference from the Sliding Window (used without the SLA option) is the fact that the state of the Alert rule remains.
By default, SLA mode is disabled. To enable it, change the SLA INTERVAL from "SLA mode not enabled" to the desired timeframe.
The following illustration shows the differences by means of a comparative analysis: