Kubernetes: an odyssey of over complexity

To play around with Kubernetes, rather than building one from scratch and having to fix a billion different things that could go wrong, I decided to start off with a completely working Kubernetes cluster. So, from https://medium.com/@raj10x/multi-node-kubernetes-cluster-with-vagrant-virtualbox-and-kubeadm-9d3eaac28b98 , I used the Vagrantfile gist at the end and ran vagrant up

and found myself, after thousands of unintelligible lines of code had scrolled past, back at the command line. Had things worked?

Did I need to be worried about the reams of gibberish output like:

k8s-head: service/calico-typha created
k8s-head: Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
k8s-head: customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
k8s-head: customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
k8s-head: customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
k8s-head: customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
k8s-head: customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
k8s-head: customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
k8s-head: customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
k8s-head: customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
k8s-head: customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
k8s-head: serviceaccount/calico-node created
k8s-head: unable to recognize "https://raw.githubusercontent.com/ecomm-integration-ballerina/kubernetes-cluster/master/calico/calico.yaml": no matches for kind "Deployment" in version "apps/v1beta1"
k8s-head: unable to recognize "https://raw.githubusercontent.com/ecomm-integration-ballerina/kubernetes-cluster/master/calico/calico.yaml": no matches for kind "DaemonSet" in version "extensions/v1beta1"
k8s-head: W0921 14:01:39.054627 6427 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
 k8s-node-2: Processing triggers for systemd (229-4ubuntu21.29) ...
 k8s-node-2: sed: can't read /etc/default/kubelet: No such file or directory
 k8s-node-2: dpkg-preconfigure: unable to re-open stdin: No such file or directory
 k8s-node-2: Warning: Permanently added '' (ECDSA) to the list of known hosts.
 k8s-node-2: [preflight] Running pre-flight checks
 k8s-node-2: [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/

Who knows?! I had no idea what the state of the cluster was. Had anything worked? Had something worked? Had nothing worked?

So, some debugging:

  1. State of VMs
vagrant status
Current machine states:

k8s-head running (virtualbox)
k8s-node-1 running (virtualbox)
k8s-node-2 running (virtualbox)

So, at least the VMs are running. That would have been a good useful thing to output!

2. State of nodes

(^^^^^ I try not to go flying off on a tangent with technology – it’s so easy to end up going down a rabbit hole of realising this is wrong and then that’s wrong however one thing that seems consistently broken across many editors is this 1. 2. numbering indent. The first indent is almost impossible to delete and the second indent is almost impossible to insert)

vagrant ssh k8s-head

kubectl get services
kubernetes ClusterIP <none> 443/TCP 23h

kubectl get nodes
k8s-head NotReady master 23h v1.19.2
k8s-node-1 NotReady <none> 23h v1.19.2
k8s-node-2 NotReady <none> 23h v1.19.2

So, the nodes don’t seem to be ready.

vagrant@k8s-head:~$ alias k=kubectl
vagrant@k8s-head:~$ k get po -o wide
No resources found in default namespace.
vagrant@k8s-head:~$ k get po -o wide --all-namespaces
kube-system coredns-f9fd979d6-4gmwp 0/1 Pending 0 23h <none> <none> <none> <none>
kube-system coredns-f9fd979d6-pgdtv 0/1 Pending 0 23h <none> <none> <none> <none>

So, something seems wrong with coredns.

Having all these namespaces just adds to the confusion / complexity. E.g.

vagrant@k8s-head:~$ k logs coredns-f9fd979d6-4gmwp
Error from server (NotFound): pods "coredns-f9fd979d6-4gmwp" not found
vagrant@k8s-head:~$ k logs coredns-f9fd979d6-4gmwp --all-namespaces
Error: unknown flag: --all-namespaces
See 'kubectl logs --help' for usage.

Getting the logs of something shouldn’t be this difficult. You shouldn’t have to pick up a book on Kubernetes or do the CKA or search StackOverflow to find out simple stuff like this.

OK, this page sounds promising: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-pod-replication-controller/

This page shows how to debug Pods and ReplicationControllers.

Let’s try their first example:

vagrant@k8s-head:~$ kubectl describe pods coredns
Error from server (NotFound): pods "coredns" not found
vagrant@k8s-head:~$ kubectl describe pods coredns-f9fd979d6-4gmwp
Error from server (NotFound): pods "coredns-f9fd979d6-4gmwp" not found

It then says:

Look at the output of the kubectl describe … command above. There should be messages from the scheduler about why it can not schedule your pod

There aren’t any messages from the scheduler so another dead end.


So, going to Slack Kubernetes Office Hours someone suggests:

kubectl -n kube-system logs coredns-f9fd979d6-4gmwp

which usefully outputs nothing at all. Literally nothing. That’s a big fat nothing useful at all. Not even a This Pod is Pending!

It finally turned out the magic invocation was:

kubectl -n kube-system describe pod coredns-f9fd979d6-4gmwp
Name: coredns-f9fd979d6-4gmwp
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: <none>
Labels: k8s-app=kube-dns
Annotations: <none>
Status: Pending
IPs: <none>
Controlled By: ReplicaSet/coredns-f9fd979d6
Image: k8s.gcr.io/coredns:1.7.0
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
memory: 170Mi
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
/etc/coredns from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from coredns-token-gv9wj (ro)
Type Status
PodScheduled False 
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
Type: Secret (a volume populated by a Secret)
SecretName: coredns-token-gv9wj
Optional: false
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 76m default-scheduler 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

and after a whole lot more gibberish it finally gets to:

Warning FailedScheduling 76m default-scheduler 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.


kubectl get nodes
k8s-head NotReady master 24h v1.19.2
k8s-node-1 NotReady <none> 24h v1.19.2
k8s-node-2 NotReady <none> 24h v1.19.2

So, coredns did not deploy because the nodes were tainted with not-ready rather than the Nodes being in a NotReady status due to coredns being pranged. i.e. the problem is with the Nodes.

Checking head:

kubectl describe node k8s-head
Name: k8s-head
Roles: master
Labels: beta.kubernetes.io/arch=amd64
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 21 Sep 2020 14:01:34 +0000
Taints: node-role.kubernetes.io/master:NoSchedule
Unschedulable: false
HolderIdentity: k8s-head
AcquireTime: <unset>
RenewTime: Tue, 22 Sep 2020 14:24:35 +0000
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 22 Sep 2020 14:22:39 +0000 Mon, 21 Sep 2020 14:01:30 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 22 Sep 2020 14:22:39 +0000 Mon, 21 Sep 2020 14:01:30 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 22 Sep 2020 14:22:39 +0000 Mon, 21 Sep 2020 14:01:30 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Tue, 22 Sep 2020 14:22:39 +0000 Mon, 21 Sep 2020 14:01:30 +0000 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Hostname: k8s-head
cpu: 2
ephemeral-storage: 10098468Ki
hugepages-2Mi: 0
memory: 2047912Ki
pods: 110
cpu: 2
ephemeral-storage: 9306748094
hugepages-2Mi: 0
memory: 1945512Ki
pods: 110
System Info:
Machine ID: e7bd138c751d41f4a33dd882c048ced4
System UUID: 33E00DBD-58B6-E844-97AF-749AD4E96AB6
Boot ID: d2d28c19-c5ef-443a-9c1d-20ebeaeb1082
Kernel Version: 4.4.0-189-generic
OS Image: Ubuntu 16.04.7 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://17.3.3
Kubelet Version: v1.19.2
Kube-Proxy Version: v1.19.2
Non-terminated Pods: (5 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system etcd-k8s-head 0 (0%) 0 (0%) 0 (0%) 0 (0%) 24h
kube-system kube-apiserver-k8s-head 250m (12%) 0 (0%) 0 (0%) 0 (0%) 24h
kube-system kube-controller-manager-k8s-head 200m (10%) 0 (0%) 0 (0%) 0 (0%) 24h
kube-system kube-proxy-k2vw9 0 (0%) 0 (0%) 0 (0%) 0 (0%) 24h
kube-system kube-scheduler-k8s-head 100m (5%) 0 (0%) 0 (0%) 0 (0%) 24h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 550m (27%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events: <none>

So, right in the middle of all that gibberish:

Ready False Tue, 22 Sep 2020 14:22:39 +0000 Mon, 21 Sep 2020 14:01:30 +0000 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

So, maybe try a different Vagrantfile after I’ve lost a day debugging this one.


Kubernetes Up & Running: Chapter 7



kubectl run alpaca-prod --image=gcr.io/kuar-demo/kuard-amd64:1 --replicas=3 --port=8080 --labels="ver=1,app=alpaca,env=prod"
kubectl expose deployment alpaca-prod
kubectl run bandicoot-prod --image=gcr.io/kuar-demo/kuard-amd64:2 --replicas=2 --port=8080 --labels="ver=2,app=bandicoot,env=prod"
kubectl expose deployment bandicoot-prod
kubectl get services -o wide


In another terminal:

ALPACA_POD=$(kubectl get pods -l app=alpaca -o jsonpath='{.items[0].metadata.name}')
echo $ALPACA_POD alpaca-prod-7f94b54866-dwwxg
kubectl port-forward $ALPACA_POD 48858:8080

Forwarding from -> 8080
Forwarding from [::1]:48858 -> 8080


Now access the cluster with:



  • use http not https
  • If you get localhost refused to connect. then check original pod. E.g.
    • kubectl logs alpaca-prod-7f94b54866-dwwxg
2019/01/09 11:44:22 Starting kuard version: v0.7.2-1
2019/01/09 11:44:22 **********************************************************************
2019/01/09 11:44:22 * WARNING: This server may expose sensitive
2019/01/09 11:44:22 * and secret information. Be careful.
2019/01/09 11:44:22 **********************************************************************
2019/01/09 11:44:22 Config:
  "address": ":8080",
  "debug": false,
  "debug-sitedata-dir": "./sitedata",
  "keygen": {
    "enable": false,
    "exit-code": 0,
    "exit-on-complete": false,
    "memq-queue": "",
    "memq-server": "",
    "num-to-gen": 0,
    "time-to-run": 0
  "liveness": {
    "fail-next": 0
  "readiness": {
    "fail-next": 0
  "tls-address": ":8443",
  "tls-dir": "/tls"
2019/01/09 11:44:22 Could not find certificates to serve TLS
2019/01/09 11:44:22 Serving on HTTP on :8080

which seems to indicate it’s successfully serving on 8080 locally.

So, the issue is with the code on Page 67. i.e. it should be:

kubectl port-forward $ALPACA_POD 8080:8080


DNS Resolver: http://localhost:8080/-/dns

with alpaca-prod


alpaca-prod.default.svc.cluster.local.	5	IN	A

i.e. name of service: alpaca-prod

namespace: default

resource type: svc

base domain: cluster.local

Note: you could use:

  • alpaca-prod.default
  • alpaca-prod.default.svc.cluster.local.


Adding in a readinessProbe:

      - image: gcr.io/kuar-demo/kuard-amd64:1
        imagePullPolicy: IfNotPresent
        name: alpaca-prod
                path: /ready
                port: 8080
            periodSeconds: 2
            initialDelaySeconds: 0
            failureThreshold: 3
            successThreshold: 1

and restart port-forward (as the pods are recreated).

There should now be a Readiness Probe tab where you can make that pod fail / succeed /ready checks.

The pod with that IP address is destroyed after 3 fails and recreated after it succeeds.

k get endpoints alpaca-prod --watch


Now, after halting the port-forward and watch, we’ll look at NodePorts:

kubectl edit service alpaca-prod

and change

type: ClusterIP


type: NodePort

It immediately changes when you save. i.e.

kubectl describe service alpaca-prod

shows `Type: NodePort`

Note: if you misspell the Service type you’ll immediately be bounced back into the Editor with a couple of lines at the top indicating the problem. E.g.

# services "alpaca-prod" was not valid:
# * spec.type: Unsupported value: "Nodey": supported values: "ClusterIP", "ExternalName", "LoadBalancer", "NodePort"




See also Kubernetes: kubectl

Kubernetes: Labels vs Annotations

What’s the difference between a Label and an Annotation?

Labels contain identifying information and used by selector queries. They are used by Kubernetes internally so, for performance reasons, there are limits on the size of labels (63 characters or less).



Annotations are used for non-identifying information, especially large and/or structured/unstructured data.




Kubernetes Up & Running: Chapter 4

Common kubectl commands

Namespace and Contexts

kubectl config set-context my-context --namespace=mystuff

kubectl config use-context my-context


Note: to list namespaces, see Kubernetes: Namespaces


Page 34

kubectl get pods

Note -o wide gives more information.

kubectl describe pod <pod id>


Page 35

kubectl label pods <pod id> color=green

Show labels:

kubectl get pods --show-labels

Remove label:

kubectl label pods bar color-

Note: the command in the book does not work:

kubectl label pods bar -color


unknown shorthand flag: 'c' in -color



Copy-and-paste: https://github.com/rusrushal13/Kubernetes-Up-and-Running/blob/master/Chapter4.md

Kubernetes Up & Running: Chapter 2

Page 16

1. make sure you’ve cloned the git repo mentioned earlier in the chapter

2. once in the repo, run:

make build

to build the application.

3. create this Dockerfile (not the one mentioned in the book)

FROM alpine
LABEL maintainer="e@snowcrash.eu"
COPY bin/1/amd64/kuard /kuard
ENTRYPOINT ["/kuard"]

MAINTAINER is deprecated. Use a LABEL instead: https://github.com/kubernetes-up-and-running/kuard/issues/7

However, whilst MAINTAINER can take 1 argument, LABELtakes key/value pairs. E.g.

LABEL <key>=<value>



And the  COPY path in the book is incorrect.

and run

docker build -t kuard-amd64:1 .

to build the Dockerfile.

Here we’ve got a repo name of kuard-amd64 and a tag of 1.


4. Check the repo using

docker images

Note: a registry is a collection of repositories, and a repository is a collection of images



Page 17

Files removed in subsequent layers are not available but still present in the image.


Image sizes:

docker images

Or a specific one:

docker image ls <repository name>

E.g. alpine is 4.41MB.


Let’s create a 1MB file and add / remove it:

dd if=/dev/zero of=1mb.file bs=1024 count=1024


then copy it in:

FROM alpine
COPY 1mb.file /

Now building it (and creating a repo called alpine_1mb) we can see the image has increased in size by a bit over 1MB (probably due to the overhead of an additional layer).

However, if we now remove this file in a subsequent Dockerfile – e.g. with something like:

FROM alpine_1mb
RUN rm /1mb.file

the image is still the same size.

The solution is to ensure you use an rm in the same RUN command as you create/use your big file: https://stackoverflow.com/questions/53998310/docker-remove-intermediate-layer

and https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#run


Page 19


To run the kuard app, use:

docker run -d --name kuard --publish 8080:8080 gcr.io/kuar-demo/kuard-amd64:1



docker tag kuard-amd64:1 gcr.io/kuar-demo/kuard-amd64:1

According to https://docs.docker.com/get-started/part2/#tag-the-image

the tag command is:

docker tag image username/repository:tag

so image is kuard-amd64:1

but what’s the username?

Is it gcr.io ?

Or gcr.io/kuar-demo?

The answer is that Docker’s docs here:


are incorrect. You don’t need a username or repository. It’s just a label. E.g. see https://docs.docker.com/engine/reference/commandline/tag/

Correct would be:

docker tag image <any label you want>:tag

BUT for the purposes of pushing to a repository that label DOES need to be of a specific format. i.e. username/image_name.


Shame they didn’t explain that in more detail.


And the next line is misleading too.

docker push gcr.io/kuar-demo/kguard-amd64:1

This creates the impression that you’re pushing your image to a website (or registry) hosted at gcr.io.

It’s not.

It’s just the tag you created above. Having said that, I had to simplify the tag (from 2 slashes to 1 slash) to get it to work. E.g.

docker tag kuard-amd64:1 snowcrasheu/kuard-amd64:1

docker push snowcrash/kuar/kuard-amd64:1
The push refers to repository [docker.io/snowcrash/kuar/kuard-amd64]
7b816b232464: Preparing
73046094a9b8: Preparing
denied: requested access to the resource is denied

The reason for

denied: requested access to the resource is denied

is that (from https://stackoverflow.com/questions/41984399/denied-requested-access-to-the-resource-is-denied-docker )

You need to include the namespace for Docker Hub to associate it with your account.
The namespace is the same as your Docker Hub account name.
You need to rename the image to YOUR_DOCKERHUB_NAME/docker-whale.



To login with docker use:

docker login

or to use a specific username / password.

docker login -u <username> -p <password>

Better is --password-stdin however.

and push with:

docker push snowcrasheu/kuard-amd64:1

which you should then be able to see on Docker Hub. E.g.



Limit CPU / Memory

docker run -d --name kuard
--publish 8080:8080
--memory 200m
--memory-swap 1G
--cpu-shares 1024 



How to change a repository name: 



Handy copy-and-paste code: https://github.com/rusrushal13/Kubernetes-Up-and-Running/blob/master/Chapter2.md