Kubernetes: creating a new namespace and using it in a new context

Here’s how to isolate your work using a new namespace called dev in a new context, also called dev:

Note: I’m aliasing kubectl to k.

List namespaces with: kubectl get namespaces --show-labels

 

And to quickly switch between contexts use kubectx:

Install:

brew install kubectx

List contexts:

kubectx

Switch:

kubectx <name>

 

Delete namespaces with:

kubectl delete namespaces <name>

Note: this will not delete it from your config file – https://stackoverflow.com/questions/53283120/kubernetes-cant-delete-namespace/53283273#53283273

 

To do this declaratively, use a config file. E.g.

https://kubernetes.io/docs/tasks/administer-cluster/namespaces/#subdividing-your-cluster-using-kubernetes-namespaces

 

See also:

 

 

Configure kubectl for Amazon EKS

To use the stock kubectl client for EKS you need to:

  • install the AWS IAM Authenticator for Kubernetes

https://docs.aws.amazon.com/eks/latest/userguide/configure-kubectl.html

  • modify your kubectl configuration file to use it for authentication

 

Other things that may be useful are:

  • helm – if you’re using Helm charts to manage your cluster in EKS
  • kubectl and awscli – goes without saying

E.g. check your aws cli version with:

aws --version and upgrade with pip install awscli --upgrade --user

  • assume-role – if you’re using IAM roles

https://github.com/remind101/assume-role

  • nice to have is fzf: https://github.com/junegunn/fzf#installation

 

To update your kubeconfig use:

aws eks update-kubeconfig --name CLUSTER_NAME-eks --region REGION

You’ll need an up-to-date version of the awscli. E.g. 1.15.53 won’t cut it.

 

To assume role use:

eval $(assume-role <role-name>)

Issues:

If you get:

it would be because you don’t have a profile in your ~/.aws/config

Your profile in ~/.aws/config should look like:

 

You should be able to run:

assume-role <role-name>

and see the assume role output.

 

 

 

Testing:

To test you can access your EKS cluster, use:

kubectl get all -n kube-system

Or for none-system:

kubectl get all

 

 

Deploy a WordPress app using Kubernetes on AWS

We’ll start off locally with minikubethen deploy to AWS.

Our guide will be https://github.com/kubernetes/examples/tree/master/mysql-wordpress-pd

Locally

git clone git@github.com:kubernetes/examples.git

Install kubectl:

brew install kubernetes-cli

https://kubernetes.io/docs/tasks/tools/install-kubectl/

 

Install minikube:

brew cask install minikube

https://github.com/kubernetes/minikube/releases

 

Install virtualbox

brew cask install virtualbox

 

minikube start

Any problems, use minikube delete.

https://github.com/kubernetes/minikube/issues/459

 

And here’s an Asciinema Asciicast of deploying WordPress on Kubernetes via minikube

The crucial commands:

Note if you’re using a namespace then you’ll need to do:

minikube service -n <your namespace> wordpress --url

https://stackoverflow.com/questions/53283657/kubernetes-could-not-find-finalized-endpoint-being-pointed-to-error-validating/53283783#53283783

 

Note: if we’d forgotten to configure the secrets then we’d have seen:

which we’d need to debug further with:

kubectl describe pods wordpress-mysql

and at the end you’d see (under Events:

Warning Failed 8s (x10 over 104s) kubelet, minikube Error: secrets "mysql-pass" not found

 

How to decode secrets

 

To list deployments use:

kubectl get deployment

or, for a specific deployment:

kubectl get deployment <your dep>

https://kubernetes.io/docs/reference/kubectl/cheatsheet/

 

See also: https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/#deploy-wordpress

Production-ready (also local)

Use Helm.

brew install kubernetes-helm

helm init

(otherwise you’ll get Error: could not find tiller)

helm install stable/wordpress

 

Check on progress with:

kubectl get pods

 

Issues:

https://stackoverflow.com/questions/53286172/error-from-server-badrequest-container-mariadb-in-pod-is-waiting-to-start

k get pods

Debugging:

k describe pods wp-1-mariadb-0

revealed:

 

kubectl: create pod

Let’s try creating a pod.

  1. create a pod.ymlwith your favourite image:

 

Then:

kubectl create -f pod.yml

returns

pod/hello-pod created

but we need to check what’s going on behind the scenes with:

kubectl get pods

Or if you have loads of pods, you can use:

kubectl get pods/hello-pod

or all pods in all namespaces:

kubectl get pods --all-namespaces

“kubectl get pods” warnings

kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-pod 0/1 ContainerCreating 0 1m

kubectl describe pods

This:

Reason: ContainerCreating

is because it’s still creating the container.

kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-pod 0/1 ImagePullBackOff 0 10m

Again:

kubectl describe pods

Reason: ImagePullBackOff

means there’s a problem with the image.

Check with:

docker inspect --type=image <name of image>:latest

I was getting:

Error: No such image:

So, let’s try another image. i.e. in pod.ymllet’s use:

image: hello-world:latest

Now create hits:

 

Let’s try deleting this pod with:

https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/

 

Let’s try --all as mentioned:

Solution is:

Note: this can take a few minutes.

https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/

 

Let’s try create again:

What is a CrashLoopBackOff? How to alert, debug / troubleshoot, and fix Kubernetes CrashLoopBackOff events.

Why am I getting this?

‘cos the hello-world image exits after printing out its introductory message.

Let’s try:

image: yeasy/simple-web:latest

We can keep track of events with:

kubectl get events

After around 5 mins it comes up. Let’s interact with the web server. First what IP does it have?

kubectl get pods -l zone=prod -o yaml | grep IP

kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-pod 1/1 Running 0 12m

kubectl exec -it hello-pod — /bin/bash

and

curl -i localhost:80

shows it’s working.

 

Note: you’d normally deploy a pod using a ReplicationController.

Delete existing pod with:

kubectl delete pods hello-pod

and create a ReplicationController with:

Note: 4 top-level objects are:

  • apiVersion
  • kind
  • metadata
  • spec

then

kubectl create -f rc.yml

And inspect with:

To increase replicas, update rc.yml and apply with:

kubectl apply -f rc.yml

 

Other errors

Unable to connect to the server

Unable to connect to the server: dial tcp 192.168.64.5:8443: i/o timeout

  • check minikube is started – minikube start

kubectl describe pods

 

 

 

Kubernetes: kubectl

kubectl lets you control the Kubernetes Cluster Manager.

https://kubernetes.io/docs/reference/kubectl/kubectl/

 

Quick tips

create – creates resources

apply – applies resources

 

Common Commands

See also the kubectl Cheat Sheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/

 

Show config: k config view

 

To switch to a non-VT cluster (e.g. minikube), need to:

kubectl config get-contexts

k config use-context minikube

 

And view cluster-info or nodes (e.g. to check) with:

Cluster info: k cluster-info

List nodes: k get nodes

Show deployments: k get deployments

List contexts: k config get-contexts

 

Pods

Get all pods: k get pods

Describe all pods: k describe pods

Describe pod: k describe pods <pod name>

Listing all pods:

export POD_NAME=$(kubectl get pods -o go-template –template ‘{{range .items}}{{.metadata.name}}{{“\n”}}{{end}}’)

echo Name of the Pod: $POD_NAME

E.g. get output from pod using proxy

k proxy

curl http://localhost:8001/api/v1/namespaces/default/pods/<name of pod>/proxy/

Logs from Pod: k logs $POD_NAME

 

Services

List services: k get services

Describe service: k describe services/kubernetes-bootcamp

Docs

See also this great tutorial: https://kubernetes.io/docs/tutorials/kubernetes-basics/explore/explore-interactive/

and Cheat Sheet: https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/

 

Command Summary

If you run kubectl without any arguments you’ll get some information about it. E.g. some commands:

 

Following examples assume we’ve got the name of the Pod in $POD_NAME, e.g. using:

export POD_NAME=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')

(for more on Go templates see Go: templates

 

Commands – details

apply

Apply additional changes.

Note: this warning can be ignore:

Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply

config view

Shows kubeconfig settings which you can query. E.g.

kubectl config view -o jsonpath='{.users[?(@.name == "minikube")].user.client-certificate}'

https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-context-and-configuration

create

Create resources

delete

Deletes resources.

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#delete

Note that deleting a service does not stop an app running inside a container.

describe

Shows details of a resource.

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe

E.g.

kubectl describe deploy <deployment name>

exec

Run commands on the container.

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#exec

E.g. to see environment variables:

kubectl exec $POD_NAME env

Or to get a terminal:

kubectl exec -it $POD_NAME bash

https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/

expose

Creates and exposes a new service to traffic.

Note: --type can be:

  • ClusterIP
  • NodePort
  • LoadBalancer
  • ExternalName.

Default is ClusterIP.

E.g. kubectl expose deployment/kubernetes-bootcamp –type=”NodePort” –port 8080

Running kubectl get services shows the new service.

List the port with describe services/kubernetes-bootcamp.

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#expose

get

Gets resources. E.g. for deployments:

get deployments

Services:

get services

Or pods:

get pods

Note: for a wider format use -o wide.

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get

Note: you can specify the output as ‘template’ and provide a Go template as the value of the --template flag, to filter the attributes of the fetched resources. E.g.

kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'

For more on Go Templates see: https://golang.org/pkg/text/template/

logs

Prints logs for a container in a pod.

Anything the application sends to STDOUT becomes the logs for the container.

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs

proxy

Creates a proxy (note: this is blocking so need to run it in a new terminal) that forwards communications into the cluster-wide private network which we can query directly using the API. E.g.

curl http://localhost:8001/version

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#proxy

rollout

Manage the rollout of a resource. E.g.

kubectl rollout status deployments/kubernetes-bootcamp

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#rollout

Or

kubectl rollout status deployment <name of deployment>

 

And view the history with:

kubectl rollout history deployment <name of deployment>

Note: to make sure rollouts are recorded use --record=true. E.g.

kubectl apply --filename=deploy.yml --record=true

run

runNAME –image=… –port=…

This:

  • searches for a node
  • schedules the application (i.e. the image) to run on that node
  • configures the cluster to the instance on a new node when needed

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#run

Notes: this is the iterative method.

scale

Sets a new size. E.g. number of pods in a deployment.

Trying this on kubernetes.io I scaled beyond capacity and got:

Unable to connect to the server: net/http: TLS handshake timeout

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#scale

set

Configure application resources – e.g. to update a deployment

See also Kubernetes: update a deployed application and rollback

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#set

 

Errors / Problems

The connection to the server 192.168.99.100:8443 was refused – did you specify the right host or port?

Did you start your Kubernetes cluster? E.g. if you’re using minikube then:

minikube start