Day 77- 90DaysOfDevOps - Grafana Alerting

Hey Learners! Welcome back. We understood the concept of Grafana and monitored the system logs locally with some security. What if the system exceeds some threshold? Grafana Alerts are a way to send notifications when a metric crosses the threshold you have configured. Let's dive deeper into it...

Task 1- Set Up Sample Alerting

Grafana Alerting allows us to create alerts and take action on alerts in a single consolidated view of our metrics and logs, no matter where they are stored. Improve our team's ability to identify and resolve issues quickly.

Steps:-

Login to Grafana. Click three lines from the upper left to see the left pane. Navigate to the Alerting section on the left pane.
We have to create an Alert rule for the free disk space. Click on Alert rules, select Prometheus as data source, State will be as Firing with Alert as Rule type and health is Ok. Click New alert rule.

Enter the alert rule name and define the query and the condition for alerting as follows
Add new Reduce expression to collect exact number of bytes and then add Math expression to convert reduced value in GB as shown. We have to add threshold value to trigger alarm. Refer following screenshot

Use query as follow. Choose device and mountpoint as per your requirement.
```
 node_filesystem_avail_bytes{device="/dev/sda2",mountpoint="/"}
```
Note:- If the final condition returns value as 1 it will trigger the alert. In this case we are adding free dick space should not be lesser than(IS BELOW) 870 (GB no need to specify) and present free disk space is about 874 GB which validates our condition false and returns expression value as 0 and hence as value is 0 alert will not triggered showing Normal state.
Create a folder and group for evaluation behaviour

Save rule and exit.
One alert rule created and showing state as Normal.
What if return value is 1? Alert goes to Firing state and will notify by the email or many other options.
To send notification we have to configure SMTP on Grafana server.
Edit /etc/grafana/grafana.ini file as shown below. Use sudo vim /etc/grafana/grafana.ini command to edit

Note:- You should enable Two Factor Authentication for your email and get App Password for the same to configure SMTP.
Restart Grafana Server using sudo systemctl restart grafana-server command.
To send an notification via email, we first have to add contact points. Go to contact points from left pane of Alerting section and click on Add contact point. Enter the name and Email address to get an notification on. Before adding, Test the configuration. After Test alert sent successfully save the contact point. Make sure you enter correct email address to get mail from configured mail address in SMTP configuration.
Now we'll set threshold to fire alert as below. Duplicate the Alert rule.
Edit the rule as below and Save rule and exit.
Wait for a minute and Alert state will change from Pending to Firing as we have less space than specified threshold.
Check the email received from configured mail.