Docker: building a secure swarm

Swarm cluster

Creates:

  • Cluster store
  • Manager
  • Mutual auth encryption
  • Worker

More info: https://docs.docker.com/engine/swarm/

SwarmKit

docker swarm init

at a node then:

  • ups it as a Manager of the Swarm
  • which makes it a Leader
  • root CA of the swarm
  • issues itself a client certificate
  • creates a Cluster store (using etcd)
  • and a certificate rotation policy
  • Mgr/Wrks encryption tokens

Now: add another Node

  • gets allocated another certificate and a join token
  • also a Manager (a Follower Manager)

If we send in a command, it gets proxied to the Leader.

If the Leader fails, another gets elected as Leader using Raft.

Manager HA best practice: Have 3, 5, or 7

Workers join in the same way: i.e. with

docker swarm join

https://docs.docker.com/engine/reference/commandline/swarm_join/

Example

Say we have a container that’s been created from an Alpine image. e.g.

docker ps -a

Start it with:

docker container start 1ff

Initialise swarm with:

docker swarm init

Note: you’d only do this once on the first node.

To check swarm mode is active use:

docker system info

NOT the obvious: docker swarm info which doesn’t exist.

Now we can list the nodes in the swarm with:

docker node ls

https://docs.docker.com/engine/reference/commandline/node_ls/

To get a token for a manager:

docker swarm join-token manager

and to join the swarm as a manager use the outputted command. e.g.

docker swarm join --token SWMTKN-1-237d4spc3iwhjfyfjjpxkjzku9ay4cqcj4saxuf79432kma124-7rvw6dp390ugfsradu6t63zcy 192.168.65.3:2377

Note: the swarm token takes the form:

Swarm Token label: SWMTKN

Cluster Identifier: 237d4spc3iwhjfyfjjpxkjzku9ay4cqcj4saxuf79432kma124(i.e. everything in the same cluster will have this ID)

Worker or Manager identifier: 7rvw6dp390ugfsradu6t63zcy(i.e. will a node be admitted as a worker or as a manager. In this case this is a Manager identifier)

Note: we haven’t created any networks yet. The default network that Docker uses is the ingress overlay network. e.g.

kqon5lut6bev ingress overlay swarm

This handles control and data traffic related to swarm services.

 

List nodes with docker node ls.

Add worker nodes with docker swarm join-token worker.

Note that a worker node can’t query the swarm with docker node ls.

Note autolock can be enabled with --autolock=true which prevents:

  • restarted managers from automatically re-joining the swarm
  • accidentally restoring old copies of the swarm

Note: to view client certificates:

sudo openssl x509 -in /var/lib/docker/swarm/certificates/swarm-node.crt -text

E.g.  Subject: O = nxglnqmu6n68bwri1h0i2x10f, OU = swarm-manager, CN = fj3hxa3l8jf0rnypxmfawthaf

0 => Organisation == Swarm ID

OU => Organisation Unit == Node’s role

CN => Canonical Name == cryptographic Node ID

 

Potential Errors

SubConns are in TransientFailure– e.g.

Error response from daemon: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = “transport: Error while dialing dial tcp 10.0.2.15:2377: connect: connection refused”

You don’t have IP connectivity.

Make sure your VMs can connect to each other. E.g. in my Vagrantfile I use:

and

i.e. I’m creating a private network for the VMs.

Then, to use this private network, use the--advertise-addroption:

docker swarm init --advertise-addr 10.0.0.10

 

Leave a Reply

Your email address will not be published. Required fields are marked *