Some core terms

An endpoint is an instance – e.g. a single process.

A collection of instances with the same purpose (e.g. a replicated process such as an API server) is called a job.

A node is a target – e.g. localhost on port 9090.

https://prometheus.io/docs/introduction/first_steps/

https://prometheus.io/docs/introduction/overview/

https://prometheus.io/docs/concepts/jobs_instances/

Configuration

Prometheus is configured via /etc/prometheus/prometheus.yml

and typically starts with:

global:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/

 

e.g. let’s dissect this:

groups:
- name: example
  rules:
  - alert: HighErrorRate
    expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5
    for: 10m
    labels:
      severity: page
    annotations:
      summary: High request latency

See alerting rules: https://github.com/prometheus/prometheus/blob/master/docs/configuration/alerting_rules.md

and recording rules: https://github.com/prometheus/prometheus/blob/master/docs/configuration/recording_rules.md

and this on notifications

https://github.com/prometheus/prometheus/blob/master/docs/configuration/alerting_rules.md#sending-alert-notifications

and this on expr:

https://pierrevincent.github.io/2017/12/prometheus-blog-series-part-5-alerting-rules/

 

Basics of querying:

1. Go to Prometheus –https://prom-server/graph

2. Enter time series selectors

e.g.
http_requests_total
or

node_filesystem_avail

or with a label

node_filesystem_avail{mountpoint="/"}

 

Notes:

Label matching operators:

• = Select labels that are exactly equal to the provided string
• != Select labels that are not equal to the provided string
• =~ Select labels that regex-match the provided string (or substring)
• !~ Select labels that do not regex-match the provided string (or substring)

 

Get list of metrics available on Prom server using:

curl http://localhost:9090/metrics

 

And targets:

curl http://localhost:9090/api/v1/targets

https://prometheus.io/docs/prometheus/latest/querying/basics/

/api/v1 is the HTTP API.

E.g. see https://prometheus.io/docs/prometheus/latest/querying/api/

More later.

 

More useful docs:

https://petargitnik.github.io/blog/2018/01/04/how-to-write-rules-for-prometheus

 

Note: Prometheus was developed to monitor web services. To monitor a node, you’ll need Node Exporter: https://www.digitalocean.com/community/tutorials/how-to-use-prometheus-to-monitor-your-centos-7-server

 

HTTP API

is exposed at /api/v1.

https://prometheus.io/docs/prometheus/latest/querying/api/

and label values:

https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values

E.g. `curl http://localhost:9090/api/v1/label/job/values`

gets all the label values for the job label.

 

Exporters

It’s the job of an exporter to export values from a node into Prometheus. E.g. on an Elasticsearch node:

ps -ef | grep export
root 11637 1 0 Mar21 ? 00:44:18 /usr/local/bin/elasticsearch_exporter -web.listen-address=:9000
root 15603 1 0 2017 ? 03:10:45 /usr/local/bin/node_exporter -web.listen-address=:10000 -collector.textfile.directory=/var/local

we can see here an Elasticsearch exporter and a node exporter (for CPU, etc metrics).

The Elasticsearch exporter is configured to send data to Prometheus as follows:

 

and we can check the data in Prometheus via:

 

 

Notes:

Marvel allows you to monitor Elasticsearch via Kibana. As of 5.0, Marvel is part of X-Pack.

https://www.elastic.co/guide/en/marvel/current/introduction.html