Prometheus: Configuration, Querying and PromQL

Some core terms

An endpoint is an instance – e.g. a single process.

A collection of instances with the same purpose (e.g. a replicated process such as an API server) is called a job.

A node is a target – e.g. localhost on port 9090.

https://prometheus.io/docs/introduction/first_steps/

https://prometheus.io/docs/introduction/overview/

https://prometheus.io/docs/concepts/jobs_instances/

Configuration

Prometheus is configured via /etc/prometheus/prometheus.yml

and typically starts with:

global:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/

 

e.g. let’s dissect this:

See alerting rules: https://github.com/prometheus/prometheus/blob/master/docs/configuration/alerting_rules.md

and recording rules: https://github.com/prometheus/prometheus/blob/master/docs/configuration/recording_rules.md

and this on notifications

https://github.com/prometheus/prometheus/blob/master/docs/configuration/alerting_rules.md#sending-alert-notifications

and this on expr:

https://pierrevincent.github.io/2017/12/prometheus-blog-series-part-5-alerting-rules/

 

Basics of querying:

1. Go to Prometheus –https://prom-server/graph

2. Enter time series selectors

e.g.
http_requests_total
or

node_filesystem_avail

or with a label

node_filesystem_avail{mountpoint="/"}

 

Notes:

Label matching operators:

  • = Select labels that are exactly equal to the provided string
  • != Select labels that are not equal to the provided string
  • =~ Select labels that regex-match the provided string (or substring)
  • !~ Select labels that do not regex-match the provided string (or substring)

 

Get list of metrics available on Prom server using:

curl http://localhost:9090/metrics

 

And targets:

curl http://localhost:9090/api/v1/targets

https://prometheus.io/docs/prometheus/latest/querying/basics/

/api/v1 is the HTTP API.

E.g. see https://prometheus.io/docs/prometheus/latest/querying/api/

More later.

 

More useful docs:

https://petargitnik.github.io/blog/2018/01/04/how-to-write-rules-for-prometheus

 

Note: Prometheus was developed to monitor web services. To monitor a node, you’ll need Node Exporter: https://www.digitalocean.com/community/tutorials/how-to-use-prometheus-to-monitor-your-centos-7-server

 

HTTP API

is exposed at /api/v1.

https://prometheus.io/docs/prometheus/latest/querying/api/

and label values:

https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values

E.g. curl http://localhost:9090/api/v1/label/job/values

gets all the label values for the job label.

 

Exporters

It’s the job of an exporter to export values from a node into Prometheus. E.g. on an Elasticsearch node:

we can see here an Elasticsearch exporter and a node exporter (for CPU, etc metrics).

The Elasticsearch exporter is configured to send data to Prometheus as follows:

 

and we can check the data in Prometheus via:

 

 

Notes:

Marvel allows you to monitor Elasticsearch via Kibana. As of 5.0, Marvel is part of X-Pack.

https://www.elastic.co/guide/en/marvel/current/introduction.html

 

Monitoring containers with Prometheus and Grafana

Architecting Monitoring for Containerized Applications

Why not use Nagios?

Can’t use same method as traditional servers. E.g. putting an agent into a container doesn’t really work.

/metrics exposed for container runtime. Docker uses Prometheus format (i.e. simple text with Key Value format)

Prometheus stores data in time series database.

Prometheus configuration

Is in YAML. E.g.

/etc/prometheus/prometheus.yml

Sections:

scrape_configs

- job_name: <name here>

and

scrape_interval: 60s

Prometheus Dashboard

Status > Targets: lists all monitored targets

Graph > Graph > select from insert metric at cursor

 

Collecting Metrics with Prometheus

Exposing Runtime Metrics with Prometheus

Exposing Application Metrics to Prometheus

Exposing Docker Metrics to Prometheus

Building Dashboards with Grafana

 

 

 

Prometheus: storage

Prometheus has its own local storage using a local on-disk time series database. However, this is not clustered or replicated. i.e. it’s not scalable or durable.

https://prometheus.io/docs/prometheus/latest/storage/

It does provide interfaces to integrate with remote storage. E.g.

https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations

and

https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage

 

One of these options is PostgreSQL and TimescaleDB. Note, TimescaleDB uses PostgreSQL but scales it for better performance using automatic partitioning across time and space:

https://github.com/timescale/prometheus-postgresql-adapter

and https://github.com/timescale/timescaledb

 

Prometheus remote storage adapter for PostgreSQL

1. Install packages (both provided by Timescale):

  • remote storage adapter

The adapter is a translation proxy used by Prometheus for reading/writing data to the PostgreSQL/Timescale database. The data from Prometheus arrives as a Protobuf. The adapter deserializes it and converts it into the Prometheus native format (see Prometheus’ Exposition Formats) before inserting it into the database.

A Docker image provides the Prometheus PostgreSQL remote storage adapter: https://hub.docker.com/r/timescale/prometheus-postgresql-adapter/

  •  pg_prometheus

pg_prometheusimplements the Prometheus data model for PostgreSQL.

A Docker image which provides PostgreSQL and TimescaleDB: https://hub.docker.com/r/timescale/pg_prometheus/

https://blog.timescale.com/sql-nosql-data-storage-for-prometheus-devops-monitoring-postgresql-timescaledb-time-series-3cde27fd1e07

2. Configure Prometheus to use this remote storage adapter

i.e. add this to prometheus.yml

 

See also this tutorial: https://docs.timescale.com/v0.10/tutorials/prometheus-adapter