Configuring Prometheus Alerts and Visualizing Them in Grafana

Monitoring your systems is crucial, but knowing when to act is equally important. In this guide, we’ll walk through setting up Prometheus alerts based on specific metric thresholds and displaying these alerts in Grafana. We’ll use a high CPU usage scenario as our example.

Prerequisites

Before we begin, ensure you have:

1. Prometheus installed and collecting metrics
2. Grafana set up and connected to Prometheus
3. Alertmanager installed (for handling alerts)

Step 1: Configuring Prometheus Alerting Rules

First, let’s create an alerting rule for high CPU usage. Create or edit a file named alert.rules.yml in your Prometheus configuration directory:

groups:
- name: cpu_alerts
  rules:
  - alert: HighCPUUsage
    expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High CPU usage detected on {{ $labels.instance }}"
      description: "CPU usage is above 80% on instance {{ $labels.instance }} for the last 5 minutes."

This rule triggers an alert when CPU usage exceeds 80% for 5 minutes.

Step 2: Updating Prometheus Configuration

Add the rules file to your Prometheus configuration (prometheus.yml):

rule_files:
  - "alert.rules.yml"
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093

Restart Prometheus to apply these changes.

Step 3: Configuring Alertmanager

Create or edit the Alertmanager configuration file (alertmanager.yml):

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'
receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://127.0.0.1:5001/'

This configuration sends alerts to a webhook. In a real-world scenario, you might configure email or Slack notifications here.

Step 4: Verifying Alerts in Prometheus

To check if your alerts are configured correctly:

1. Open the Prometheus web interface (usually at http://localhost:9090)
2. Navigate to the “Alerts” tab
3. You should see your “HighCPUUsage” alert listed

Step 5: Displaying Alerts in Grafana

Now, let’s set up Grafana to display these alerts:

1. Log into your Grafana dashboard
2. Create a new dashboard or edit an existing one
3. Add a new panel
4. In the query editor, switch to the “Alert list” visualization
5. Under the “Alert list” tab, configure the following:

• Data source: Prometheus
• Alert name filter: HighCPUUsage
• State filter: Alerting, Pending
• Max items: 10 (or your preferred number)

6. Save the panel and dashboard

Step 6: Creating a Grafana Alert

While Prometheus handles the alerting logic, we can also create a visual alert in Grafana:

1. Edit your CPU usage panel in Grafana
2. Go to the “Alert” tab
3. Click “Create Alert”
4. Set conditions (e.g., “WHEN last() OF query(A, 5m, now) IS ABOVE 80”)
5. Set the evaluation interval and duration
6. Add notifications if desired
7. Save the alert

Testing Your Setup

To test your alert:

1. Artificially increase CPU usage on a monitored system
2. Wait for the alert duration (5 minutes in our example)
3. Check the Prometheus UI to see if the alert fires
4. Verify that the alert appears in your Grafana dashboard

Conclusion

You’ve now set up a complete alerting system:

1. Prometheus monitors your systems and evaluates alert conditions
2. Alertmanager handles alert routing and notifications
3. Grafana provides a visual representation of your alerts

This setup allows you to:

• Quickly identify when systems exceed important thresholds
• Visualize alert status alongside your metrics
• Respond promptly to potential issues

Remember to fine-tune your alert thresholds and durations based on your specific needs and system behaviors. Regular review and adjustment of your alerting rules will help maintain an effective monitoring system.

So, whether you’re a tech enthusiast, a professional, or just someone who wants to learn more, I invite you to follow me on this journey. Subscribe to my blog and follow me on social media to stay in the loop and never miss a post.

Together, let’s explore the exciting world of technology and all it offers. I can’t wait to connect with you!”

Connect me on Social Media: https://linktr.ee/mdshamsfiroz

Happy coding! Happy learning!

Happy monitoring!