Project - Kubernetes Cluster Monitoring with Grafana and Prometheus

Hey Learners! Welcome back. In this blog we'll learn how to monitor K8s cluster with the help of Prometheus and Grafana. Let's start...

Prerequisite:-

K8s Cluster (I am using Minikube Cluster)
Helm Chart
Grafana and Prometheus should be installed (Installed Grafana and Prometheus on Same node running Minikube Cluster)

Steps:-

Grafana and Prometheus Installation
Node Exporter Installation
Kube-State-Metrics Deployment
Prometheus and Node Exporter integration
Prometheus and Kube-State-Metrics integration
Creating Dashboards

Step 1- Grafana and Prometheus Installation

As the part of 90DaysOfDevOps challenge we already covered installation of Grafana. Click here to know more about Grafana installation.

Grafana is a popular open-source data visualisation and analytics platform that allows you to create custom dashboards and visualisations based on a variety of data sources. Grafana is often used for monitoring and analysing metrics and logs in real-time, making it an ideal tool for monitoring systems and applications, including Kubernetes environments.

Prometheus is an open-source monitoring and alerting system that helps you collect and store metrics about your software systems and infrastructure, and analyse that data to gain insights into their health and performance. It provides a powerful query language, a flexible data model, and a range of integrations with other tools and systems. With Prometheus, you can easily monitor metrics such as CPU usage, memory usage, network traffic, and application-specific metrics, and use that data to troubleshoot issues, optimise performance, and create alerts to notify you when things go wrong.

To install Prometheus follow the below steps

sudo useradd --system --no-create-home --shell /bin/false prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.47.1/prometheus-2.47.1.linux-amd64.tar.gz
tar -xvf prometheus-2.47.1.linux-amd64.tar.gz
cd prometheus-2.47.1.linux-amd64/
sudo mkdir -p /data /etc/prometheus
sudo mv prometheus promtool /usr/local/bin/
sudo mv consoles/ console_libraries/ /etc/prometheus/
sudo mv prometheus.yml /etc/prometheus/prometheus.yml
sudo chown -R prometheus:prometheus /etc/prometheus/ /data/

Know we have to create service for Prometheus

sudo vim /etc/systemd/system/prometheus.service command and enter the below details and save.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/data \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-lifecycle

[Install]
WantedBy=multi-user.target

Use following commands to enable and start Prometheus service

sudo systemctl enable prometheus and sudo systemctl start prometheus

Check status of Prometheus service and it should be in running state using sudo systemctl status prometheus

Access prometheus using <localhost:9090>

Step 2- Node Exporter Installation

Node Exporter is a Prometheus exporter specifically designed to gather system-level metrics from a target machine.

Follow the below steps

sudo useradd --system --no-create-home --shell /bin/false node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
rm -rf node_exporter*

sudo vim /etc/systemd/system/node_exporter.service and edit as follows and save the file.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter --collector.logind

[Install]
WantedBy=multi-user.target

Use sudo systemctl enable node_exporter command to enable the service.

Start the service using sudo systemctl start node_exporter command and access it using <localhost:9100>

Step 3- Kube-State-Metrics Deployment

Kube-State-Metrics is a service that talks to the Kubernetes API server to get all the details about all the API objects like deployments, pods, daemon sets, Statefulsets, etc

We can deploy Kube-State-Metrics service using Helm Charts or directly deployed with some Kubernetes objects like service account, service, service role, cluster role, cluster role binding, etc.

Use this link to deploy Kube-State-Metrics service directly. If this will not work try my GitHub_Repo. Preferred way to deploy in production if you need to use different nodes for Grafana and Prometheus.

I am using Helm chart to deploy Kube-State-Metrics service. Follow below steps to for Helm chart

Add Helm Repo using helm repo add prometheus-community https://prometheus-community.github.io/helm-charts command then update the helm repository using helm repo update command.
Install the Prometheus objects using helm command as helm install prometheus prometheus-community/prometheus .

Get details of running pods using kubectl get pods command. Pod for Kube-State-Metrics service is running.

Note:- There are Two containers running with Prometheus Server which are used to configure with Grafana server but as we already installed Grafana we'll not expose that Prometheus Server Pod for our use.
We deploy Kube-State-Metrics service successfully. Now we need to expose Kube-State-Metrics service as it only accessible within the cluster we need to expose it for external world using NodePort service.

Use the kubectl expose service prometheus-kube-state-metrics --type=NodePort --target-port=8080 --name=kubestateservice command to expose kube-state service to others instead of cluster.

Use kubectl get svc command to check which node port is assigned for the kube-state service.

Here I have port 30468 assigned for kube-state service.
Use <minikubeIP:NodePort> to access through web.

Choose Metrics to see all gathered metrics for k8s cluster.

We have successfully get all the metrics of K8s cluster with kube-state service.

Step 4- Prometheus and Node Exporter integration

We need to integrate Node exporter with Prometheus server so that Prometheus will store logs from node exporter and sends to Grafana server for visualisation.

Edit the prometheus.yml file as follow. Use sudo vim /etc/prometheus/prometheus.yml command to edit.

Note:- As we are using same node for prometheus and node exporter use localhost:PortNumber as targets.

After making any changes to prometheus.yml file use curl -X POST http://localhost:9090/-/reload/ command to reflect on web UI.

Step 5- Prometheus and Kube-State-Metrics integration

Same as before we integrate Node exporter with Prometheus we have to add kube-state service's URL as a target in promtheus.yml file as below. Note- use minikube IP instead of localhost with node port.

Use curl _X POST http://localhost:9090/-/reload/ command and have web access using http://localhost:9090 URL.

Step 6- Creating Dashboards

We will create dashboard as shown below for Node Details

We'll see how to create dashboard for CPU utilisation and for other

For other panels copy the JSON format from GitHub.

For Minikube Custer Dashboard will look like below. Import JSON file from the same Git Repo as mentioned above.

Note:- Import dashboard to check how Prometheus query used for collecting metrics and then try in your environment with making some changes accordingly.

Thank you so much for taking the time to read till the end! Hope you found this blog informative.

Feel free to explore more of my content, and don't hesitate to reach out if need any assistance from me or in case of you have any questions.

Find me on:- Hashnode LinkedIn Github

Happy Learning!