What are the steps to set up a monitoring system using Prometheus and Grafana?

12 June 2024

In today's rapidly changing technological landscape, the ability to monitor systems in real-time is crucial. This article will guide you step by step on how to set up an effective monitoring system using Prometheus and Grafana. Whether you are managing a small server or a large Kubernetes cluster, understanding the nuances of these powerful tools can greatly enhance your ability to maintain and optimize your systems.

Before diving into the setup process, it is essential to understand what Prometheus and Grafana are, and why they are widely used in the industry.

Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It collects and stores metrics as time series data, records real-time metrics in a time series database, and uses a powerful query language called PromQL to retrieve data.

Grafana, on the other hand, is an open-source data visualization and monitoring platform. It allows you to create dynamic dashboards and graphs, making it easier to interpret complex data collected by Prometheus.

The combination of Prometheus and Grafana forms a robust system for monitoring infrastructure and applications, ensuring you can keep an eye on critical metrics and respond to issues before they escalate.

Step 1: Installing Prometheus

The first step in setting up your monitoring system is to install Prometheus on your server. This guide assumes you are using a Linux AMD system.

  1. Download Prometheus:
    Begin by downloading the latest version of Prometheus from the official Prometheus website. Choose the appropriate package for your operating system.

    wget https://github.com/prometheus/prometheus/releases/download/v2.30.3/prometheus-2.30.3.linux-amd64.tar.gz
  2. Extract the Tar File:
    Extract the downloaded tar file to a suitable directory.

    tar xvfz prometheus-*.tar.gz
  3. Move Binary Files:
    Move the Prometheus binaries to the /usr/local/bin directory for easy access.

    sudo mv prometheus-2.30.3.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.30.3.linux-amd64/promtool /usr/local/bin/
  4. Create Configuration File:
    Create a configuration file named prometheus.yml in /etc/prometheus/.

    sudo mkdir -p /etc/prometheus
sudo nano /etc/prometheus/prometheus.yml

    Paste the following sample configuration into prometheus.yml:

    global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  5. Set Up Prometheus Service:
    Create a service file for Prometheus to manage it with systemctl.

    sudo nano /etc/systemd/system/prometheus.service

    Paste the following configuration into the service file:

    [Unit]
Description=Prometheus Service
After=network.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml

[Install]
WantedBy=multi-user.target
  6. Start Prometheus:
    Enable and start the Prometheus service.

    sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

Step 2: Configuring Node Exporter

To monitor system metrics such as CPU usage, memory, and disk space, you need to install and configure Node Exporter.

  1. Download Node Exporter:
    Download the latest version of Node Exporter from the official release page.

    wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz
  2. Extract the Tar File:
    Extract the downloaded tar file.

    tar xvfz node_exporter-*.tar.gz
  3. Move Binary Files:
    Move the Node Exporter binary to /usr/local/bin.

    sudo mv node_exporter-1.2.2.linux-amd64/node_exporter /usr/local/bin/
  4. Set Up Node Exporter Service:
    Create a service file for Node Exporter.

    sudo nano /etc/systemd/system/node_exporter.service

    Paste the following configuration:

    [Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
  5. Start Node Exporter:
    Enable and start Node Exporter.

    sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
  6. Prometheus Configuration:
    Update the Prometheus configuration file to include Node Exporter.

    scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

    Restart Prometheus to apply the new configuration.

    sudo systemctl restart prometheus

Step 3: Installing Grafana

Next, let's set up Grafana to visualize the metrics collected by Prometheus.

  1. Add Grafana Repository:
    Add the Grafana APT repository.

    sudo apt-get install -y software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
  2. Install Grafana:
    Update the package list and install Grafana.

    sudo apt-get update
sudo apt-get install grafana
  3. Start Grafana:
    Enable and start the Grafana service.

    sudo systemctl enable grafana-server
sudo systemctl start grafana-server
  4. Access Grafana:
    Open your web browser and navigate to http://localhost:3000. Log in with the default credentials (admin / admin).
  5. Add Prometheus Data Source:
    In Grafana, go to Configuration > Data Sources > Add data source. Select Prometheus and set the URL to http://localhost:9090. Click Save & Test to verify the connection.

Step 4: Creating Grafana Dashboards

With Grafana set up, you can now create dashboards to visualize the data collected by Prometheus.

  1. Import a Dashboard:
    You can start by importing a pre-built dashboard. Go to Dashboard > Manage > Import. Enter a dashboard ID (e.g., 1860 for Node Exporter Full) or upload a JSON file.
  2. Customize Your Dashboard:
    Customize the dashboard according to your needs. You can add or remove panels, change visualization types, and set alert rules.
  3. Create Custom Panels:
    To create a custom panel, click Add Panel > Add Query. Select Prometheus as the data source and write your query in PromQL. Adjust the visualization settings to display the data as desired.
  4. Set Up Alerts:
    Set up alerts to notify you of critical changes in your systems. Go to Alerting > Alert Rules and create a new alert rule based on a Prometheus query.

Step 5: Monitoring Kubernetes Cluster

If you're managing a Kubernetes cluster, integrating Prometheus and Grafana can provide deep insights into cluster performance.

  1. Install Prometheus in Kubernetes:
    Use Helm to install Prometheus.

    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/prometheus
  2. Install Grafana in Kubernetes:
    Install Grafana using Helm.

    helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana
  3. Configure Prometheus to Scrape Kubernetes Metrics:
    Add the Kubernetes API server and services as scrape targets in the Prometheus configuration file.
  4. Create Kubernetes Dashboards:
    Import Kubernetes-specific dashboards in Grafana, such as the Kubernetes Cluster Monitoring (ID: 315).

By following these steps, you will have successfully set up a monitoring system using Prometheus and Grafana. This system will collect, store, and visualize metrics from your servers or Kubernetes clusters, allowing you to keep track of performance and system health in real-time. Proper monitoring is vital for maintaining the stability and efficiency of your infrastructure, and with Prometheus and Grafana, you have powerful tools at your disposal.

Copyright 2024. All Rights Reserved