AWS Billing Alerts

Here’s how to get an alert if your AWS Bill exceeds a certain amount.

After you’ve enabled Billing Alerts:

1. change Region to us-east-1 (N. Virginia)

2. CloudWatch > Alarms > Create Alarm

3. Select Metric > Billing > Total Estimated Charge

4. Tick USD | EstimatedCharges

5. click Select Metric

6. Enter value in exceed

7. pick from send a notification to: (I use a List called NotifyMe with my email address). It seems to take a while for the drop down to respond.

8. click Show Advanced Option (at bottom) then update Name, Description

Note: you cannot change the Name after you’ve created an Alarm. The only way to do so is to delete your Alarm and start again.

9. click Create Alarm

 

Top tip:

  • go create alarms up to 100x your current billing in suitable increments – one day you’ll hit them and you want to be warned!

 

 

 

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html

 

Kubernetes: Service Accounts

A service account provides an identity for processes that run in a Pod.

 

e.g. if you access the cluster using kubectlyou’re authenticated by apiserver as a user account (e.g. admin).

Processes in containers also contact apiserver and are authenticated (e.g. if you don’t specify an account then it’s assigned default).

 

Check pod service account name via:

kubectl get pods/podname -o yaml

and see spec.serviceAccountName

 

List service accounts:

There doesn’t seem to be a way to view them via the Kubernetes Dashboard.

 

 

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

 

 

Service Meshes on Kubernetes: Istio, Linkerd, SuperGloo

Quick note: there’s a lot going on in the Service Mesh space for Kubernetes.

Istio (based on Envoy) is the elephant in the room with a ton of funding.

But there’s also Linkerd and SuperGloo.

And a recent announcement from AWS: AWS App Mesh.

 

Great summary of Istio:

Generally traffic is defined as north/south (into and out of the datacenter) or east/west (between servers in the datacenter).

Istio is for east/west traffic within your K8S cluster, designed to connect your services together by moving all the network traffic through the Envoy proxy. It is usually done by wrapping your deployments with an extra sidecar pod (automatically using K8S APIs) that intercepts all the networking to other services and pods. You would still use a load balancer or ingress to route external traffic into the cluster, although there are options like Heptio Contour that also use Envoy for this.

This provides a single data and control plane to centralize all network reliability, security, service discovery, and monitoring.

Note: Istio uses an extended version of the Envoy proxy: https://istio.io/docs/concepts/what-is-istio/#envoy
Istio provides:
  • Dynamic service discovery
  • Load balancing
  • TLS termination
  • HTTP/2 and gRPC proxies
  • Circuit breakers
  • Health checks
  • Staged rollouts with %-based traffic split
  • Fault injection
  • Rich metrics
And an interesting post about Service Meshes:

Prometheus: Configuration, Querying and PromQL

Some core terms

An endpoint is an instance – e.g. a single process.

A collection of instances with the same purpose (e.g. a replicated process such as an API server) is called a job.

A node is a target – e.g. localhost on port 9090.

https://prometheus.io/docs/introduction/first_steps/

https://prometheus.io/docs/introduction/overview/

https://prometheus.io/docs/concepts/jobs_instances/

Configuration

Prometheus is configured via /etc/prometheus/prometheus.yml

and typically starts with:

global:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/

 

e.g. let’s dissect this:

See alerting rules: https://github.com/prometheus/prometheus/blob/master/docs/configuration/alerting_rules.md

and recording rules: https://github.com/prometheus/prometheus/blob/master/docs/configuration/recording_rules.md

and this on notifications

https://github.com/prometheus/prometheus/blob/master/docs/configuration/alerting_rules.md#sending-alert-notifications

and this on expr:

https://pierrevincent.github.io/2017/12/prometheus-blog-series-part-5-alerting-rules/

 

Basics of querying:

1. Go to Prometheus –https://prom-server/graph

2. Enter time series selectors

e.g.
http_requests_total
or

node_filesystem_avail

or with a label

node_filesystem_avail{mountpoint="/"}

 

Notes:

Label matching operators:

  • = Select labels that are exactly equal to the provided string
  • != Select labels that are not equal to the provided string
  • =~ Select labels that regex-match the provided string (or substring)
  • !~ Select labels that do not regex-match the provided string (or substring)

 

Get list of metrics available on Prom server using:

curl http://localhost:9090/metrics

 

And targets:

curl http://localhost:9090/api/v1/targets

https://prometheus.io/docs/prometheus/latest/querying/basics/

/api/v1 is the HTTP API.

E.g. see https://prometheus.io/docs/prometheus/latest/querying/api/

More later.

 

More useful docs:

https://petargitnik.github.io/blog/2018/01/04/how-to-write-rules-for-prometheus

 

Note: Prometheus was developed to monitor web services. To monitor a node, you’ll need Node Exporter: https://www.digitalocean.com/community/tutorials/how-to-use-prometheus-to-monitor-your-centos-7-server

 

HTTP API

is exposed at /api/v1.

https://prometheus.io/docs/prometheus/latest/querying/api/

and label values:

https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values

E.g. curl http://localhost:9090/api/v1/label/job/values

gets all the label values for the job label.

 

Exporters

It’s the job of an exporter to export values from a node into Prometheus. E.g. on an Elasticsearch node:

we can see here an Elasticsearch exporter and a node exporter (for CPU, etc metrics).

The Elasticsearch exporter is configured to send data to Prometheus as follows:

 

and we can check the data in Prometheus via:

 

 

Notes:

Marvel allows you to monitor Elasticsearch via Kibana. As of 5.0, Marvel is part of X-Pack.

https://www.elastic.co/guide/en/marvel/current/introduction.html