Monitoring containers with Prometheus and Grafana

Architecting Monitoring for Containerized Applications

Why not use Nagios?

Can’t use same method as traditional servers. E.g. putting an agent into a container doesn’t really work.

/metrics exposed for container runtime. Docker uses Prometheus format (i.e. simple text with Key Value format)

Prometheus stores data in time series database.

Prometheus configuration

Is in YAML. E.g.

/etc/prometheus/prometheus.yml

Sections:

scrape_configs

- job_name: <name here>

and

scrape_interval: 60s

Prometheus Dashboard

Status > Targets: lists all monitored targets

Graph > Graph > select from insert metric at cursor

 

Collecting Metrics with Prometheus

Exposing Runtime Metrics with Prometheus

Exposing Application Metrics to Prometheus

Exposing Docker Metrics to Prometheus

Building Dashboards with Grafana

 

 

 

Prometheus: storage

Prometheus has its own local storage using a local on-disk time series database. However, this is not clustered or replicated. i.e. it’s not scalable or durable.

https://prometheus.io/docs/prometheus/latest/storage/

It does provide interfaces to integrate with remote storage. E.g.

https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations

and

https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage

 

One of these options is PostgreSQL and TimescaleDB. Note, TimescaleDB uses PostgreSQL but scales it for better performance using automatic partitioning across time and space:

https://github.com/timescale/prometheus-postgresql-adapter

and https://github.com/timescale/timescaledb

 

Prometheus remote storage adapter for PostgreSQL

1. Install packages (both provided by Timescale):

  • remote storage adapter

The adapter is a translation proxy used by Prometheus for reading/writing data to the PostgreSQL/Timescale database. The data from Prometheus arrives as a Protobuf. The adapter deserializes it and converts it into the Prometheus native format (see Prometheus’ Exposition Formats) before inserting it into the database.

A Docker image provides the Prometheus PostgreSQL remote storage adapter: https://hub.docker.com/r/timescale/prometheus-postgresql-adapter/

  •  pg_prometheus

pg_prometheusimplements the Prometheus data model for PostgreSQL.

A Docker image which provides PostgreSQL and TimescaleDB: https://hub.docker.com/r/timescale/pg_prometheus/

https://blog.timescale.com/sql-nosql-data-storage-for-prometheus-devops-monitoring-postgresql-timescaledb-time-series-3cde27fd1e07

2. Configure Prometheus to use this remote storage adapter

i.e. add this to prometheus.yml

 

See also this tutorial: https://docs.timescale.com/v0.10/tutorials/prometheus-adapter