c2b_default

Kubernetes

Karteien für die Kubernetes Fachbegriffe

Karteien für die Kubernetes Fachbegriffe

23
0.0 (0)
A. S.
Lernen
A. S.
Diese Lernkarten vermitteln fortgeschrittene Kubernetes-Kenntnisse für Studierende und IT-Profis. Sie decken zentrale Komponenten wie Pods, Nodes, den Controlplane und Services ab und erklären praktische Aufgaben wie Backups, Zertifikatsverwaltung, Logging sowie das Debuggen von Kubelet und etcd. Der Fokus liegt auf administrativen Befehlen, Sicherheitsaspekten und der Fehlersuche in komplexen Cluster-Umgebungen. Ideal für alle, die Kubernetes in Produktionsumgebungen einsetzen und dessen interne Mechanismen vertiefen möchten.
Karten
23
Lernende
3
Sprache
Deutsch
Kategorie
Informatik
Stufe
Universität
Erstellt / Aktualisiert
29.04.2024 / 03.06.2026
  • 1  / 23

Lernkarten

How can I find the CIDR of a service?

  •  ssh cluster1-controlplane1
  • root@cluster1-controlplane1:~# cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep range

Which Networking (or CNI Plugin) is configured and where is its config file?

  • root@cluster1-controlplane1:~# find /etc/cni/net.d/
  •  root@cluster1-controlplane1:~# cat /etc/cni/net.d/10-weave.conflist

Which suffix will static pods have that run on cluster1-node1 ?

The suffix is the node hostname with a leading hyphen. It used to be -static in earlier Kubernetes versions.

$do

--dry-run=client -o yaml

Write a second command doing the same thing into /opt/course/1/context_default_no_kubectl.sh , but without the use of
kubectl .

cat ~/.kube/config | grep current | sed -e "s/current-context: //"

Make a Pod that should be scheduled on controlplane nodes

spec:
   tolerations
   - effect: NoSchedule
       key: node-role.kubernetes.io/control-plane 
   nodeSelector:
       node-role.kubernetes.io/control-plane: ""

Set livnessProbe and readinessProbe

spec:
   containers:
   -   image: nginx:1.16.1-alpine
       name: ready-if-service-ready
    resources: {}
    livenessProbe:
       exec:
          command:
         - 'true'
    readinessProbe:
       exec:
         command:
         - sh
         - -c
         - 'wget -T2 -O- http://service-am-i-ready:80'

The metrics-server has been installed in the cluster. Your college would like to know the kubectl commands to:
1. show Nodes resource usage
2. show Pods and their containers resource usage

1. k top node

2. k top pod --containers

Check how the controlplane component kubelet, kube-apiserver, kube-scheduler, kube-controller-manager and etcd are started/installed on the controlplane node.

We could start by finding processes of the requested components, especially the kubelet at first:

  • ps aux | grep kubelet # shows kubelet process

We can see which components are controlled via systemd looking at /usr/lib/systemd directory:

  • find /usr/lib/systemd | grep kube
  • find /usr/lib/systemd | grep etcd

This shows kubelet is controlled via systemd, but no other service named kube nor etcd. It seems that this cluster has been setup using
kubeadm, so we check in the default manifests directory:

  • find /etc/kubernetes/manifests/

This means the main 4 controlplane services are setup as static Pods. Actually, let's check all Pods running on in the kube-system Namespace
on the controlplane node:

  • kubectl -n kube-system get pod -o wide | grep controlplane1

There we see the 5 static pods, with -cluster1-controlplane1 as suffix. We also see that the dns application seems to be coredns, but how is it controlled?

  • kubectl -n kube-system get ds
  • kubectl -n kube-system get deploy

Seems like coredns is controlled via a Deployment. We combine our findings in the requested file:

# /opt/course/8/controlplane-components.txt
kubelet: process
kube-apiserver: static-pod
kube-scheduler: static-pod
kube-controller-manager: static-pod
etcd: static-pod
dns: pod coredns

Stop the kube-scheduler, this means in a way that you can start it again afterwards.

  • cd /etc/kubernetes/manifests/
  • mv kube-scheduler.yaml ..

Manually schedule that Pod on node cluster2-controlplane1

spec:
   nodeName: cluster2-controlplane1
   containers: ...

Make the Pods of a DaemonSet run on all nodes, also controlplanes.

template:
    metadata:
       ...
    spec:
        containers:
        - image: httpd:2.4-alpine
          name: ds-important
          resources:
              requests:
                  cpu: 10m
                  memory: 10Mi
       tolerations:
       -  effect: NoSchedule 
          key: node-role.kubernetes.io/control-plane

Kill the containerd container of the kube-proxy Pod on node cluster2-node1 and write the events into /opt/course/15/
container_kill.log .

  • ssh cluster2-node1
  • crictl ps | grep kube-proxy
    • 1e020b43c4423 36c4ebbc9d979 About an hour ago Running kube-proxy
  • crictl rm 1e020b43c4423
  • k get events -A --sort-by=.metadata.creationTimestamp > /opt/course/15/container_kill.log

1. Write the names of all namespaced Kubernetes resources (like Pod, Secret, ConfigMap...) into /opt/course/16/resources.txt .

2. Find the project-* Namespace with the highest number of Roles defined in it and write its name and amount of Roles into /opt/course/16/crowded-namespace.txt .

1. k api-resources --namespaced -o name > /opt/course/16/resources.txt

2. k get namespaces

➜ k -n project-c13 get role --no-headers | wc -l
No resources found in project-c13 namespace.
0
➜ k -n project-c14 get role --no-headers | wc -l
300
...

Create a Pod named tigers-reunite . Ssh into that node and find the containerd container belonging to that Pod.

Using command crictl :
1. Write the ID of the container and the info.runtimeType into /opt/course/17/pod-container.txt
2. Write the logs of the container into /opt/course/17/pod-container.log

1. 

  • crictl ps | grep tigers-reunite
    • b01edbe6f89ed 54b0995a63052 5 seconds ago Running tigers-reunite ...
  • crictl inspect b01edbe6f89ed | grep runtimeType
    • "runtimeType": "io.containerd.runc.v2",

2. ssh cluster1-node2 'crictl logs b01edbe6f89ed' &> /opt/course/17/pod-container.log

There seems to be an issue with the kubelet not running on cluster3-node1 . Fix it and confirm that cluster has node cluster3-node1
available in Ready state afterwards. You should be able to schedule a Pod on cluster3-node1 afterwards.

First we check if the kubelet is running:

  • ssh cluster3-node1
  • ps aux | grep kubelet

Nope, so we check if it's configured using systemd as service:

  • service kubelet status

Yes, it's configured as a service with config at /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf , but we see it's inactive. Let's
try to start it:

  • service kubelet start
  • service kubelet status

We see it's trying to execute /usr/local/bin/kubelet with some parameters defined in its service config file. A good way to find errors and
get more logs is to run the command manually (usually also with its parameters).

  • whereis kubelet
    • kubelet: /usr/bin/kubelet

A wrong path was specified. Correct the path in file /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
and run:

  • vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf # fix binary path
  • systemctl daemon-reload
  • service kubelet restart

Your coworker said node cluster3-node2 is running an older Kubernetes version and is not even part of the cluster. Update Kubernetes on that node to the exact version that's running on cluster3-controlplane1 . 

Upgrade node:

  • ssh cluster3-node2
  • kubeadm upgrade node
  • apt update
  • apt show kubectl -a | grep 1.30
  • apt install kubectl=1.30.1-1.1 kubelet=1.30.1-1.1
  • kubelet --version
  • service kubelet restart
  • service kubelet status

 

Check how long the kube-apiserver server certificate is valid on cluster2-controlplane1 . Do this with openssl or cfssl. Write the exipiration
date into /opt/course/22/expiration

  • ssh cluster2-controlplane1
  • find /etc/kubernetes/pki | grep apiserver
  • openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt | grep Validity -A2

Run the correct kubeadm command to list the expiration dates and confirm both methods show the same date.

kubeadm certs check-expiration | grep apiserver

Write the correct kubeadm command that would renew the apiserver server certificate into /opt/course/22/kubeadm-renew-certs.sh .

kubeadm certs renew apiserver

Make a backup of etcd running on cluster3-controlplane1 and save it on the controlplane node at /tmp/etcd-backup.db.

  • ssh cluster3-controlplane1
  • cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd
  • ETCDCTL_API=3 etcdctl snapshot save /tmp/etcd-backup.db \
    --cacert /etc/kubernetes/pki/etcd/ca.crt \
    --cert /etc/kubernetes/pki/etcd/server.crt \
    --key /etc/kubernetes/pki/etcd/server.key

Restore the etcd backup

Stop all controlplane components:

  • cd /etc/kubernetes/manifests/
  • /etc/kubernetes/manifests# mv * ..

Restore Backup:

  • ETCDCTL_API=3 etcdctl snapshot restore /tmp/etcd-backup.db \
    --data-dir /var/lib/etcd-backup \
    --cacert /etc/kubernetes/pki/etcd/ca.crt \
    --cert /etc/kubernetes/pki/etcd/server.crt \
    --key /etc/kubernetes/pki/etcd/server.key

The restored files are located at the new folder /var/lib/etcd-backup , now we have to tell etcd to use that directory:

  • vim /etc/kubernetes/etcd.yaml
  • - hostPath: path: /var/lib/etcd-backup

Now we move all controlplane yaml again into the manifest directory. Give it some time (up to several minutes) for etcd to restart and for the
api-server to be reachable again:

  • mv ../*.yaml .
  • watch crictl ps

 

Node cluster2-node1 has been added to the cluster using kubeadm and TLS bootstrapping.
Find the "Issuer" and "Extended Key Usage" values of the cluster2-node1 :

  1. kubelet client certificate, the one used for outgoing connections to the kube-apiserver.
  2. kubelet server certificate, the one used for incoming connections from the kube-apiserver.

Client Certificate

  • openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet-client-current.pem | grep Issuer
    • Issuer: CN = kubernetes
  • openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet-client-current.pem | grep "Extended Key Usage" -A1
    • X509v3 Extended Key Usage: TLS Web Client Authentication

Server Certificate

  • penssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet.crt | grep Issuer
    • Issuer: CN = cluster2-node1-ca@1588186506
  • openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet.crt | grep "Extended Key Usage" -A1
    • X509v3 Extended Key Usage: TLS Web Server Authentication

Lernen