remember word
- taint cordon/uncordon
containerd arch layout
cmd runtime( runc, crun)
daemon runtime( containerd+ctr ) How to work with container images using ctr
image builder(buildah) Use Buildah to Build OCI Container Images
daemon image builder(buildkit+buildctl)
image regitry server(zot, docker regstry)
image pull/push(skopeo)
single mgr(nerdctl, docker)
cluster mgr(kubernetes+kubectl)
network(flannel, calico, )
network service mesh(linkerd2,)
https://github.com/kubeedge/kubeedge huawei k8s (cloud edge, iot extend) opensource item
kubernates port useage
| port | module | desc |
|---|---|---|
| 10250 | kubelet | kubelet api |
| 10248 | kubelet | health check:healthz |
| 10255 | kubelet | kubelet readonly info api,can use without auth |
| 4194 | cAdvisor |
kubernates extends
- operator Mygame is a sample kubernetes opeartor to manage 2048 game, contains crd, controller and webhook
kubernates system oper
- check ## memory section in NotesLinux.md
- container images list: kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .status.images[*]}{"\t"}{.names[0]}{"\n"}{end}{end}'
- check system memory: cat /proc/meminfo | grep '^(MemFree:|Cached:|Shmem:|Buffers:|SReclaimable|Slab:)'
- root cgroup check: cat /sys/fs/cgroup/memory.stat| grep '^(cache |rss |inactive_file |active_file |SReclaimable|Slab:)'
- cgroup memory check
alias cgcheck='function cgcheck_fn() (cd /sys/fs/cgroup/$(cat /proc/$(pgrep ${1})/cgroup | sed "s@^[^/]*/@@") && echo pid:$(pgrep ${1}) && pwd && echo "can:cat memory.stat,exit,ls,..." && bash ); cgcheck_fn'- cgcreate and use by docker
- cgcreate -g memory:test-docker-memory
- docker run --cgroup-parent=/test-docker-memory --net=none -v /root/test_mem:/test -idt --name test --privileged csighub.tencentyun.com/admin/tlinux2.2-bridge-tcloud-underlay:latest
- systemd sysemd
- systemctl == systemctl list-units
- systemctl -t slice
- systemctl -t service,masked
- systemctl list-unit-files
- systemd-cgls
- systemd-cgtop
control-plane-node-communication
- kubeadm缺省安装的已经设置好了 kubelet_node 通过证书信任 apiserver的client, 也就是这个文档中所的需要配置的地方 #只是新版本部署的anonymous-auth=false，client-ca-file这些配置都按照推荐都放到/var/lib/kubelet/config.yaml文件中了
- 如下配置让apiserver通过证书信任kubelet节点
- 生成统一的/etc/kubernetes/pki/kubelet-ca.{crt,key}
- apiserver --kubelet-certificate-authority /etc/kubernetes/pki/kubelet-ca.crt #
- 所有的node上的原有的selfsign(/var/lib/kubelet/pki/kubelet.{crt,key})替换为由 /etc/kubernetes/pki/kubelet-ca.{crt,key}签发的新crt,key
memory free >
echo 3 > /proc/sys/vm/drop_caches && swapoff -a && swapon -a && printf ‘\n%s\n’ ‘Ram-cache and Swap Cleared’
certificate how to
kubeadm certs check-expiration kubeadm certs renew all #verify cert chain openssl verify -CAfile /etc/kubernetes/pki/etcd/{ca.crt,ca.crt,healthcheck-client.crt,peer.crt,server.crt} #etcd-ca openssl verify -CAfile /etc/kubernetes/pki/{front-proxy-ca.crt,front-proxy-ca.crt,front-proxy-client.crt} #front-proxy-ca openssl verify -CAfile /etc/kubernetes/pki/{ca.crt,ca.crt,apiserver.crt,apiserver-kubelet-client.crt} openssl verify -CAfile /var/lib/kubelet/pki/kubelet.crt{,}
control pane certiciate(ca.{crt,key} etcd-ca.{crt,key} front-proxy-ca.{crt,key} sa.key sa.pub)
- upload to kube secret "kubeadm-certs"
- upload is temp, will expire in 2 hour. this is use for add more control pane
- kubeadm init phase upload-certs --upload-certs --certificate-key $(kubeadm certs certificate-key)
- kubeadm init phase upload-certs --upload-certs #Upload control plane certificates to the kubeadm-certs Secret, and print out certificate-key
- --certificate-key #define key by user option the encrypt key for uploaded cert. if not define. will auto generate new one and print out
- can check it by: kubectl -n kube-system get secret kubeadm-certs -o yaml
#regenerate apiserver.crt with new certSANs item #mv /etc/kubernetes/pki/apiserver.{crt,key} /tmp/ kubeadm init phase certs apiserver #this ok for most case kubeadm init phase certs apiserver --config <(kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}'| sed 's@^apiServer:@&\n certSANs:\n - "ctrlpe.local"@')
#check certSAN for apiserver.crt openssl x509 -in=/etc/kubernetes/pki/apiserver.crt -text | grep -A 1 "Subject Alternative Name"
kube-apiserver url acess kubelet
- kube-apiserver --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-certificate-authority=
- why kubelet serve cert need a certSANs(host list)
openssl x509 -req -days 365 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem echo subjectAltName = IP:worker_node_ip >? hostSANs_file.cnf openssl x509 -req -days 365 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile hostSANs_file.cnf
kubernetes debug utils
- journalctl -xe --unit kubelet
- kubectl describe nodes
- kubectl get nodes
- kubectl -n kube-public get cm cluster-info -o yaml
- kubectl -n kube-system get cm kubeadm-config -o yaml
- kubectl cluster-info dump
- find ip define in kubernetes> grep -r '.[0-9]{1,3}\/[0-9]{1,2}' /var/lib/kubelet/ /etc/kubernetes/ kubectl logs podName | tail -f /var/log/pods/kube-system_coredns-5dd5756b68-5wp4j_456bf805-a764-4874-b168-3c832d21241b/coredns/161.log
install k8s deps
- systemd service file ref: github:kubespray: *.service.j2
- let control-pane can get pod task(action like work pane): kubectl taint nodes nodeName1 node-role.kubernetes.io/control-plane:NoSchedule-
install k8s
install k8s network plan
- pod ip range need be diff form service ip range
- --pod-network-cidr @ kubeadm init --help
- podSubnet @/etc/kubernetes/kubeadm-config.yaml
- cluster-cidr: @/etc/kubernetes/kubeadm-config.yaml
- clusterCIDR @/etc/kubernetes/kubeadm-config.yaml
- --service-cidr @ kubeadm init --help #(default "10.96.0.0/12")
- serviceSubnet @/etc/kubernetes/kubeadm-config.yaml
- serviceSubnet @kubectl -n kube-system get cm kubeadm-config -o yaml
- --service-cluster-ip-range @/etc/kubernetes/manifests/kube-apiserver.yaml
- --pod-network-cidr @ kubeadm init --help
mkdir -p -m 755 /etc/apt/keyrings sudo apt-get update sudo apt-get install -y apt-transport-https ca-certificates curl curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl systemctl stop kubelet
config file list(ls /var/lib/kubelet/ /etc/kubernetes/manifests/ /etc/kubernetes /etc/cni/net.d/)
/etc/kubernetes:
admin.conf calico-crb.yml controller-manager.conf kubeadm-config.yaml manifests
calico-ipamconfig.yml kubeadm-images.yaml kubelet.env node-crb.yml ssl
calico-config.yml calico-node-sa.yml k8s-cluster-critical-pc.yml kubelet-config.yaml kubernetes-services-endpoint.yml pki tmp
calico-cr.yml calico-node.yml kdd-crds.yml kubelet.conf kubescheduler-config.yaml scheduler.conf
/etc/kubernetes/manifests/:
etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml
/var/lib/kubelet/:
config.yaml cpu_manager_state device-plugins kubeadm-flags.env memory_manager_state pki plugins plugins_registry pod-resources pods
/etc/cni/net.d/:
10-calico.conflist calico-kubeconfig calico.conflist.template nerdctl-bridge.conflist nerdctl-dkfile_default.conflist nerdctl-prometheus-pushgateway_default.conflist nerdctl-rtorrent_default.conflist
cat /etc/cni/net.d/10-containerd-net.conflist { "cniVersion": "1.0.0", "name": "containerd-net", "plugins": [ { "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "promiscMode": true, "ipam": { "type": "host-local", "ranges": [ [{ "subnet": "10.244.0.0/16" }], [{ "subnet": "2001:4860:4860::/64" }] ], "routes": [ { "dst": "0.0.0.0/0" }, { "dst": "::/0" } ] } }, { "type": "portmap", "capabilities": {"portMappings": true} } ] }
- grep -r 'kind: .' /etc/kubernetes/ | sort | uniq
/etc/kubernetes/admin.conf:kind: Config
/etc/kubernetes/calico-config.yml:kind: ConfigMap
/etc/kubernetes/calico-cr.yml:kind: ClusterRole
/etc/kubernetes/calico-crb.yml: kind: ClusterRole
/etc/kubernetes/calico-crb.yml:- kind: ServiceAccount
/etc/kubernetes/calico-crb.yml:kind: ClusterRoleBinding
/etc/kubernetes/calico-ipamconfig.yml:kind: IPAMConfig
/etc/kubernetes/calico-node-sa.yml:kind: ServiceAccount
/etc/kubernetes/calico-node.yml:kind: DaemonSet
/etc/kubernetes/controller-manager.conf:kind: Config
/etc/kubernetes/k8s-cluster-critical-pc.yml:kind: PriorityClass
/etc/kubernetes/kdd-crds.yml: kind: ""
/etc/kubernetes/kdd-crds.yml: kind: BGPConfiguration
/etc/kubernetes/kdd-crds.yml: kind: BGPPeer
/etc/kubernetes/kdd-crds.yml: kind: BlockAffinity
/etc/kubernetes/kdd-crds.yml: kind: CalicoNodeStatus
/etc/kubernetes/kdd-crds.yml: kind: ClusterInformation
/etc/kubernetes/kdd-crds.yml: kind: FelixConfiguration
/etc/kubernetes/kdd-crds.yml: kind: GlobalNetworkPolicy
/etc/kubernetes/kdd-crds.yml: kind: GlobalNetworkSet
/etc/kubernetes/kdd-crds.yml: kind: HostEndpoint
/etc/kubernetes/kdd-crds.yml: kind: IPAMBlock
/etc/kubernetes/kdd-crds.yml: kind: IPAMConfig
/etc/kubernetes/kdd-crds.yml: kind: IPAMHandle
/etc/kubernetes/kdd-crds.yml: kind: IPPool
/etc/kubernetes/kdd-crds.yml: kind: IPReservation
/etc/kubernetes/kdd-crds.yml: kind: KubeControllersConfiguration
/etc/kubernetes/kdd-crds.yml: kind: NetworkPolicy
/etc/kubernetes/kdd-crds.yml: kind: NetworkSet
/etc/kubernetes/kdd-crds.yml:kind: CustomResourceDefinition
/etc/kubernetes/kubeadm-config.yaml:kind: ClusterConfiguration
/etc/kubernetes/kubeadm-config.yaml:kind: InitConfiguration
/etc/kubernetes/kubeadm-config.yaml:kind: KubeProxyConfiguration
/etc/kubernetes/kubeadm-config.yaml:kind: KubeletConfiguration
/etc/kubernetes/kubeadm-images.yaml:kind: ClusterConfiguration
/etc/kubernetes/kubeadm-images.yaml:kind: InitConfiguration
/etc/kubernetes/kubelet-config.yaml:kind: KubeletConfiguration
/etc/kubernetes/kubelet.conf:kind: Config
/etc/kubernetes/kubernetes-services-endpoint.yml:kind: ConfigMap
/etc/kubernetes/kubescheduler-config.yaml:kind: KubeSchedulerConfiguration
/etc/kubernetes/manifests/etcd.yaml:kind: Pod
/etc/kubernetes/manifests/kube-apiserver.yaml:kind: Pod
/etc/kubernetes/manifests/kube-controller-manager.yaml:kind: Pod
/etc/kubernetes/manifests/kube-scheduler.yaml:kind: Pod
/etc/kubernetes/node-crb.yml: kind: ClusterRole
/etc/kubernetes/node-crb.yml: kind: Group
/etc/kubernetes/node-crb.yml:kind: ClusterRoleBinding
/etc/kubernetes/scheduler.conf:kind: Config
- kubectl get cm -n kube-system
coredns
extension-apiserver-authentication
kube-apiserver-legacy-service-account-token-tracking
kube-proxy #KubeProxyConfiguration
kube-root-ca.crt
kubeadm-config #ClusterConfiguration
kubelet-config #KubeletConfiguration
kubectl
- kubectl [get|describe|edit|set resources] (pods,rc,services,cm,deployments,rs,runtimeclass|lease) -A
kubelet_reconfig
- to learn more
- kubectl edit cm -n kube-system kubelet-config
- /var/lib/kubelet/config.yaml
- systemctl restart kubelet
k8s usage
kubectl apply -f https://kubernetes.io/examples/controllers/frontend.yaml kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml kubectl apply -f https://kubernetes.io/examples/pods/pod-rs.yaml kubectl apply -f https://k8s.io/examples/controllers/nginx-deployment.yaml
k8s config detail
kubeadm config(k8s_init_cfg&k8s_cluster_cfg helper)
- kubeadm config images list(use k8s_init_cfg gen k8s_cluster_cfg and get docker images(need use) list print out)
- kubeadm config images pull
- kubeadm config print [init-defefault|join-defaults|reset-default]
kubeadm kubeconfig(kube_access_cfg helper)
kube_access_cfg(--kubeconfig): include info(CACert,APIServer,ClientName,TokenAuth,ClientCertAuth)
- load kube_access_cfg: kubeadm&kubectl default load it from: ~/.kube/config or /etc/kubernates/admin.conf
- save kube_access_cfg when boot init: kubeadm init will generate /etc/kubernates/admin.conf at 'phase kubeconfig', reference kubeadm init -h
- send&cp /etc/kubernates/admin.conf to some user's ~/.kube/config, will give full acesss the k8s cluster to this user
- generate kube_access_cfg(with limit acesss& validity-perio) print to stdout by cmd: kubeadm kubeconfig user [--config k8s_cluster_cfg.yml]
- (kubeadm config|kubeadm upgrade|...) can use --kubeconfig kube_access_cfg.yml option to load kube_access_cfg
k8s_init_cfg(--config):
- show default k8s_init_cfg: kubeadm config print init-defaults
- save default k8s_init_cfg: kubeadm config print init-defaults > k8s_init_cfg.yml
- on other machine k8s init by: kubeadm init --config k8s_init_cfg.yml
- 'kubeadm init' == 'kubeadm init --config <(kubeadm config print init-defaults)', but you can edit output of 'kubeadm config print init-defaults' for custom init
k8s_cluster_cfg:
- show current k8s_cluster_cfg: kubectl describe -n kube-system cm kubeadm-config
kubelet config:
- [kubeadm init|join|upgrade] will write KubeletConfiguration to file /var/lib/kubelet/config.yaml and passing it to the local node kubelet #ref
- kubelet all node update: ref: Update the cgroup driver on all nodes
- kubectl drain [node-name] --ignore-daemonsets
- kubectl uncordon [node-name]