Kubernetes: an odyssey of over complexity

To play around with Kubernetes, rather than building one from scratch and having to fix a billion different things that could go wrong, I decided to start off with a completely working Kubernetes cluster. So, from https://medium.com/@raj10x/multi-node-kubernetes-cluster-with-vagrant-virtualbox-and-kubeadm-9d3eaac28b98 , I used the Vagrantfile gist at the end and ran vagrant up

and found myself, after thousands of unintelligible lines of code had scrolled past, back at the command line. Had things worked?

Did I need to be worried about the reams of gibberish output like:

Who knows?! I had no idea what the state of the cluster was. Had anything worked? Had something worked? Had nothing worked?

So, some debugging:

  1. State of VMs

So, at least the VMs are running. That would have been a good useful thing to output!

2. State of nodes

(^^^^^ I try not to go flying off on a tangent with technology – it’s so easy to end up going down a rabbit hole of realising this is wrong and then that’s wrong however one thing that seems consistently broken across many editors is this 1. 2. numbering indent. The first indent is almost impossible to delete and the second indent is almost impossible to insert)

So, the nodes don’t seem to be ready.

So, something seems wrong with coredns.

Having all these namespaces just adds to the confusion / complexity. E.g.

Getting the logs of something shouldn’t be this difficult. You shouldn’t have to pick up a book on Kubernetes or do the CKA or search StackOverflow to find out simple stuff like this.

OK, this page sounds promising: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-pod-replication-controller/

This page shows how to debug Pods and ReplicationControllers.

Let’s try their first example:

It then says:

Look at the output of the kubectl describe … command above. There should be messages from the scheduler about why it can not schedule your pod

There aren’t any messages from the scheduler so another dead end.


So, going to Slack Kubernetes Office Hours someone suggests:

kubectl -n kube-system logs coredns-f9fd979d6-4gmwp

which usefully outputs nothing at all. Literally nothing. That’s a big fat nothing useful at all. Not even a This Pod is Pending!

It finally turned out the magic invocation was:

and after a whole lot more gibberish it finally gets to:


So, coredns did not deploy because the nodes were tainted with not-ready rather than the Nodes being in a NotReady status due to coredns being pranged. i.e. the problem is with the Nodes.

Checking head:

So, right in the middle of all that gibberish:

Ready False Tue, 22 Sep 2020 14:22:39 +0000 Mon, 21 Sep 2020 14:01:30 +0000 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

So, maybe try a different Vagrantfile after I’ve lost a day debugging this one.


Kubernetes Up & Running: Chapter 7




In another terminal:


Now access the cluster with:



  • use http not https
  • If you get localhost refused to connect. then check original pod. E.g.
    • kubectl logs alpaca-prod-7f94b54866-dwwxg

which seems to indicate it’s successfully serving on 8080 locally.

So, the issue is with the code on Page 67. i.e. it should be:

kubectl port-forward $ALPACA_POD 8080:8080


DNS Resolver: http://localhost:8080/-/dns

with alpaca-prod


i.e. name of service: alpaca-prod

namespace: default

resource type: svc

base domain: cluster.local

Note: you could use:

  • alpaca-prod.default
  • alpaca-prod.default.svc.cluster.local.


Adding in a readinessProbe:

and restart port-forward (as the pods are recreated).

There should now be a Readiness Probe tab where you can make that pod fail / succeed /ready checks.

The pod with that IP address is destroyed after 3 fails and recreated after it succeeds.

k get endpoints alpaca-prod --watch


Now, after halting the port-forward and watch, we’ll look at NodePorts:

kubectl edit service alpaca-prod

and change

type: ClusterIP


type: NodePort

It immediately changes when you save. i.e.

kubectl describe service alpaca-prod

shows Type: NodePort

Note: if you misspell the Service type you’ll immediately be bounced back into the Editor with a couple of lines at the top indicating the problem. E.g.




See also Kubernetes: kubectl

Kubernetes: Labels vs Annotations

What’s the difference between a Label and an Annotation?

Labels contain identifying information and used by selector queries. They are used by Kubernetes internally so, for performance reasons, there are limits on the size of labels (63 characters or less).



Annotations are used for non-identifying information, especially large and/or structured/unstructured data.




Kubernetes Up & Running: Chapter 4

Common kubectl commands

Namespace and Contexts


Note: to list namespaces, see Kubernetes: Namespaces


Page 34

kubectl get pods

Note -o wide gives more information.

kubectl describe pod <pod id>


Page 35

kubectl label pods <pod id> color=green

Show labels:

kubectl get pods --show-labels

Remove label:

kubectl label pods bar color-

Note: the command in the book does not work:

kubectl label pods bar -color


unknown shorthand flag: 'c' in -color



Copy-and-paste: https://github.com/rusrushal13/Kubernetes-Up-and-Running/blob/master/Chapter4.md

Kubernetes Up & Running: Chapter 2

Page 16

1. make sure you’ve cloned the git repo mentioned earlier in the chapter

2. once in the repo, run:

make build

to build the application.

3. create this Dockerfile (not the one mentioned in the book)

MAINTAINER is deprecated. Use a LABEL instead: https://github.com/kubernetes-up-and-running/kuard/issues/7

However, whilst MAINTAINER can take 1 argument, LABELtakes key/value pairs. E.g.

LABEL <key>=<value>



And the  COPY path in the book is incorrect.

and run

docker build -t kuard-amd64:1 .

to build the Dockerfile.

Here we’ve got a repo name of kuard-amd64 and a tag of 1.


4. Check the repo using

docker images

Note: a registry is a collection of repositories, and a repository is a collection of images



Page 17

Files removed in subsequent layers are not available but still present in the image.


Image sizes:

docker images

Or a specific one:

docker image ls <repository name>

E.g. alpine is 4.41MB.


Let’s create a 1MB file and add / remove it:

dd if=/dev/zero of=1mb.file bs=1024 count=1024


then copy it in:

Now building it (and creating a repo called alpine_1mb) we can see the image has increased in size by a bit over 1MB (probably due to the overhead of an additional layer).

However, if we now remove this file in a subsequent Dockerfile – e.g. with something like:

the image is still the same size.

The solution is to ensure you use an rm in the same RUN command as you create/use your big file: https://stackoverflow.com/questions/53998310/docker-remove-intermediate-layer

and https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#run


Page 19


To run the kuard app, use:

docker run -d --name kuard --publish 8080:8080 gcr.io/kuar-demo/kuard-amd64:1



docker tag kuard-amd64:1 gcr.io/kuar-demo/kuard-amd64:1

According to https://docs.docker.com/get-started/part2/#tag-the-image

the tag command is:

docker tag image username/repository:tag

so image is kuard-amd64:1

but what’s the username?

Is it gcr.io ?

Or gcr.io/kuar-demo?

The answer is that Docker’s docs here:


are incorrect. You don’t need a username or repository. It’s just a label. E.g. see https://docs.docker.com/engine/reference/commandline/tag/

Correct would be:

docker tag image <any label you want>:tag

BUT for the purposes of pushing to a repository that label DOES need to be of a specific format. i.e. username/image_name.


Shame they didn’t explain that in more detail.


And the next line is misleading too.

docker push gcr.io/kuar-demo/kguard-amd64:1

This creates the impression that you’re pushing your image to a website (or registry) hosted at gcr.io.

It’s not.

It’s just the tag you created above. Having said that, I had to simplify the tag (from 2 slashes to 1 slash) to get it to work. E.g.

docker tag kuard-amd64:1 snowcrasheu/kuard-amd64:1

The reason for

denied: requested access to the resource is denied

is that (from https://stackoverflow.com/questions/41984399/denied-requested-access-to-the-resource-is-denied-docker )



To login with docker use:

docker login

or to use a specific username / password.

docker login -u <username> -p <password>

Better is --password-stdin however.

and push with:

docker push snowcrasheu/kuard-amd64:1

which you should then be able to see on Docker Hub. E.g.



Limit CPU / Memory



How to change a repository name: 



Handy copy-and-paste code: https://github.com/rusrushal13/Kubernetes-Up-and-Running/blob/master/Chapter2.md