Kubernetes: an odyssey of over complexity

To play around with Kubernetes, rather than building one from scratch and having to fix a billion different things that could go wrong, I decided to start off with a completely working Kubernetes cluster. So, from https://medium.com/@raj10x/multi-node-kubernetes-cluster-with-vagrant-virtualbox-and-kubeadm-9d3eaac28b98 , I used the Vagrantfile gist at the end and ran vagrant up

and found myself, after thousands of unintelligible lines of code had scrolled past, back at the command line. Had things worked?

Did I need to be worried about the reams of gibberish output like:

Who knows?! I had no idea what the state of the cluster was. Had anything worked? Had something worked? Had nothing worked?

So, some debugging:

  1. State of VMs

So, at least the VMs are running. That would have been a good useful thing to output!

2. State of nodes

(^^^^^ I try not to go flying off on a tangent with technology – it’s so easy to end up going down a rabbit hole of realising this is wrong and then that’s wrong however one thing that seems consistently broken across many editors is this 1. 2. numbering indent. The first indent is almost impossible to delete and the second indent is almost impossible to insert)

So, the nodes don’t seem to be ready.

So, something seems wrong with coredns.

Having all these namespaces just adds to the confusion / complexity. E.g.

Getting the logs of something shouldn’t be this difficult. You shouldn’t have to pick up a book on Kubernetes or do the CKA or search StackOverflow to find out simple stuff like this.

OK, this page sounds promising: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-pod-replication-controller/

This page shows how to debug Pods and ReplicationControllers.

Let’s try their first example:

It then says:

Look at the output of the kubectl describe … command above. There should be messages from the scheduler about why it can not schedule your pod

There aren’t any messages from the scheduler so another dead end.


So, going to Slack Kubernetes Office Hours someone suggests:

kubectl -n kube-system logs coredns-f9fd979d6-4gmwp

which usefully outputs nothing at all. Literally nothing. That’s a big fat nothing useful at all. Not even a This Pod is Pending!

It finally turned out the magic invocation was:

and after a whole lot more gibberish it finally gets to:


So, coredns did not deploy because the nodes were tainted with not-ready rather than the Nodes being in a NotReady status due to coredns being pranged. i.e. the problem is with the Nodes.

Checking head:

So, right in the middle of all that gibberish:

Ready False Tue, 22 Sep 2020 14:22:39 +0000 Mon, 21 Sep 2020 14:01:30 +0000 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

So, maybe try a different Vagrantfile after I’ve lost a day debugging this one.


git: Git Flow vs Trunk Based Development

Git Flow:

  • problems:
    • long-lived feature branches and merge hell (i.e. merge conflicts with other people’s code): https://stxnext.com/blog/2018/02/28/escape-merge-hell-why-i-prefer-trunk-based-development-over-feature-branching-and-gitflow/
    • database migrations – e.g. a db migration living in a long-lived feature branch

Trunk Based Development: release from tagged branches off master

  • short-lived branch and merge – one core rule: deploy a new commit to trunk every day
  • revert commit (if something ends up in production that you don’t want)
  • to avoid delivering code of an unfinished feature: branch by abstraction, feature flag
  • one button rollbacks (i.e. in CI/CD pipeline)
  • automated smoke tests in production that automatically roll back code if it fails a test
  • commit directly to master


git: remove secrets from history

Let’s say you’ve accidentally added a password into git. Here’s how you remove it:

1. if you have only committed locally

a. and it’s your last commit

  • just edit the file and run git commit -a --amend

b. and it’s a previous commit

  • do an interactive rebase with git rebase -i origin/master
  • change the pick to edit where you want to edit
  • amend the commit with git commit --amend
  • and continue with git rebase --continue


2. if you have committed and pushed to GitHub



Passive aggressive handovers

Possibly one of the worst handovers I’ve ever experienced. I overheard this recently when a consultant was doing a handover to a colleague. Some classics!

  • It’s all self-documenting code. It’s obvious
  • Oh, you need to run it in a separate profile to get it to work, Choose the Debug Profile
  • Come on, this is obvious. Don’t embarrass yourself by asking
  • You’ve asked three times already. Don’t embarrass yourself by asking
  • One moment, we’ll get there (in response to a question – and then never returning to the question)
  • Dunno if I’ve the energy
  • Will my batteries on my laptop survive? (when at 100%)

I’m not making this up. These were all real phrases!

And then there were all the passive-aggressive phrases like “Yes” or “No” in response to questions instead of elaborating and explaining.

And the git commit history was funny. Line after line of commit messages saying “WIP”! And all the merges to master were done by himself – no peer reviews.

You can’t make this stuff up!


git and Sublime Text integration

I’ve tried GitSavvy and struggled using it.

I like Sublime Merge.

Install with Command Palette (Ctrl Shift P), Install, Sublimemerge 3.

Then use via Command Palette. E.g. blame > Sublime Merge: Blame File will launch a new window showing tons of detail.

Also, if you’re wanting Blame integration, take a look at Blame Explorer. Once installed just hover over the line number in a file to see the change log.