Day 75 & 76- 90DaysOfDevOps

Hey Learners! Welcome back. In the previous challenge, we understood the basic concepts of Loki and Promtail to integrate with Grafana. As we have a basic idea of how to monitor logs generated on a local machine but what about the containers running? Moving forward in this challenge, we'll understand the concept of monitoring containers using the cAdvisor tool and Prometheus with Grafana. Let's start...

Prerequisite:-

• EC2 instance with Docker and Docker-Compose with appropriate permissions.

• Some running containers(at least 1)

• Grafana installed on EC2 instance

Steps:-

1. Run the docker container for Prometheus and cAdvisor with docker-compose

2. Integrate cAdvisor with Prometheus

3. Expose the cAdvisor and Prometheus

4. Integrate Prometheus with Grafana server

5. Monitor Logs for containers and create Dashboards(Day 76 challenge)

Step 1- Run the docker container for Prometheus and cAdvisor with docker-compose

We need to create a docker-compose file as follows. Use the vim docker-compose.yml command

version: '3.2'
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
    - 9090:9090
    command:
    - --config.file=/etc/prometheus/prometheus.yml
    volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    depends_on:
    - cadvisor
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.47.2
    container_name: cadvisor
    ports:
    - 8080:8080
    volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro
        devices:
          - /dev/kmsg:/dev/kmsg
        privileged: true
    depends_on:
    - redis
  redis:
    image: redis:latest
    container_name: redis
    ports:
    - 6379:6379

Step2- Integrate cAdvisor with Prometheus

cAdvisor, short for Container Advisor, is an open-source tool used for monitoring containers.

Prometheus is an open-source technology used for monitoring and alerting functionality. It can collect and store metrics as time series data, recording information with a timespan.

Before going to run docker-compose command, create a configuration file for Prometheus as follows to integrate with cAdvisor.

scrape_configs:
- job_name: cadvisor
  scrape_interval: 5s
  static_configs:
  - targets:
    - cadvisor:8080

Use the docker-compose up -d option to create and run containers

Step 3- Expose the cAdvisor and Prometheus

To expose cAdvisor and Prometheus for basic understanding we need to allow ports (9090 for Prometheus and 8080 for cAdvisor) in the Security Group

Use publicIP:8080 to access the web console for cAdvisor where you will find all the details like usage, memory, filesystem, etc of containers present in the system

Now check whether the targets are added in Prometheus or not.

Access web UI using publicIP:9090

Done. We integrate cAdvisor with Prometheus.

Step 4- Integrate Prometheus with Grafana server

Login to Grafana Web console using publicIP:3000

Go to Data Sources search for Prometheus and select it to add. Use http://localhost:9090 as a server URL. Refer DAYLINK challenge

Step 5(Day76 Challenge)- Monitor Logs for containers and create Dashboards

Click on Create Dashboards and Add panel as a virtualization. Select Prometheus as a data source and add Query as follows for the CPU of a particular container.

Apply the changes and click on the Upper right 3 dots to duplicate the panel. Refer to the below screenshot. (Change container accordingly)

Refer to GitHub and import the dashboard

Add new matric of Network receive for both App1 and App2 as shown below. To see metrics try to hit the app1/app2 webpage continuously.

Note- The above screenshot shows metrics for the last 1 Hour.

To see metrics from the Last 5 min change it from the upper right side

Thank you so much for taking the time to read till the end! Hope you found this blog informative.

Feel free to explore more of my content, and don't hesitate to reach out if need any assistance from me or in case of you have any questions.

Find me on:- Hashnode LinkedIn Github

Happy Learning!