前言
k8s 组件 环境准备
创建目录关闭防火墙关闭selinux关闭swap开启内核模块
分发到所有节点启用systemd自动加载模块服务 配置系统参数
分发到所有节点加载系统参数 清空iptables规则配置 PATH 变量下载二进制文件 部署 master 节点
创建 ca 根证书部署 etcd 组件
创建 etcd 证书配置 etcd 为 systemctl 管理分发证书以及创建相关路径启动 etcd 服务 部署 apiserver 组件
创建 apiserver 证书创建 metrics-server 证书配置 apiserver 为 systemctl 管理分发证书以及创建相关路径启动 apiserver 服务配置 kubectl 管理
创建 admin 证书创建 kubeconfig 证书分发 kubeconfig 证书到所有 master 节点 部署 controller-manager 组件
创建 controller-manager 证书创建 kubeconfig 证书配置 controller-manager 为 systemctl 管理分发证书以及创建相关路径启动 controller-manager 服务 部署 scheduler 组件
创建 scheduler 证书创建 kubeconfig 证书配置 scheduler 为 systemctl 管理分发证书以及创建相关路径启动 scheduler 服务 部署 work 节点
部署 containerd 组件
下载二进制文件配置 containerd 为 systemctl 管理配置 containerd 配置文件配置 crictl 管理工具配置 cni 网络插件分发配置文件以及创建相关路径启动 containerd 服务导入 pause 镜像 部署 kubelet 组件
创建 kubelet 证书创建 kubeconfig 证书配置 kubelet 配置文件配置 kubelet 为 systemctl 管理分发证书以及创建相关路径启动 kubelet 服务查看节点是否 Ready 部署 proxy 组件
创建 proxy 证书创建 kubeconfig 证书配置 kube-proxy 配置文件配置 proxy 为 systemctl 管理分发证书以及创建相关路径启动 kube-proxy 服务 部署 flannel 组件
配置 flannel yaml 文件配置 flannel cni 网卡配置文件导入 flannel 镜像分发 flannel cni 网卡配置文件在 k8s 中运行 flannel 组件检查 flannel pod 是否运行成功 部署 coredns 组件
配置 coredns yaml 文件导入 coredns 镜像在 k8s 中运行 flannel 组件检查 flannel pod 是否运行成功 部署 metrics-server 组件
配置 metrics-server yaml 文件导入 metrics-server 镜像在 k8s 中运行 metrics-server 组件检查 metrics-server pod 是否运行成功 部署 dashboard 组件
配置 dashboard yaml 文件导入 dashboard 镜像在 k8s 中运行 dashboard 组件检查 dashboard pod 是否运行成功查看 dashboard 访问端口查看 dashboard 登录 token 前言
为什么用 containerd ？
因为 k8s 早在2021年就说要取消 docker-shim ，相关的资料可以查看下面的链接
弃用 Dockershim 的常见问题
k8s 组件迟早都要接受的，不如早点接受
Kubernetes 组件
master 节点 组件名称 组件作用 etcd兼具一致性和高可用性的键值数据库，可以作为保存 Kubernetes 所有集群数据的后台数据库。kube-apiserver提供了资源操作的唯一入口，各组件协调者，并提供认证、授权、访问控制、API注册和发现等机制；
以 HTTP API 提供接口服务，所有对象资源的增删改查和监听操作都交给 apiserver 处理后再提交给 etcd 存储。kube-controller-manager负责维护集群的状态，比如故障检测、自动扩展、滚动更新等；
处理集群中常规后台任务，一个资源对应一个控制器，而 controllermanager 就是负责管理这些控制器的。kube-scheduler负责资源的调度，按照预定的调度策略将 pod 调度到相应的机器上。
work 节点 组件名称 组件作用 kubeletkubelet 是 master 在 work 节点上的 agent，管理本机运行容器的生命周期，比如创建容器、pod 挂载数据卷、下载 secret 、获取容器和节点状态等工作。
kubelet 将每个 pod 转换成一组容器。负责维护容器的生命周期，同时也负责 volume（CVI）和网络（CNI）的管理kube-proxy负责为 service 提供 cluster 内部的服务发现和负载均衡；
在 work 节点上实现 pod 网络代理，维护网络规则和四层负载均衡工作。container runtime负责镜像管理以及Pod和容器的真正运行（CRI）
目前用的比较多的有 docker 、 containerdcluster networking集群网络系统
目前用的比较多的有 flannel 、calicocoredns负责为整个集群提供DNS服务ingress controller为服务提供外网入口metrics-server提供资源监控dashboard提供 GUI 界面环境准备 IP 角色 内核版本 192.168.91.19master/workcentos7.6/3.10.0-957.el7.x86_64192.168.91.20workcentos7.6/3.10.0-957.el7.x86_64 service version etcdv3.5.1kubernetesv1.23.3cfsslv1.6.1containerdv1.5.9pausev3.6flannelv0.15.1corednsv1.8.6metrics-serverv0.5.2dashboardv2.4.0
cfssl github
etcd github
k8s github
containerd github
runc github
本次部署用到的安装包和镜像都上传到csdn了
master节点的配置不能小于2c2g，work节点可以给1c1g
创建目录节点之间需要完成免密操作，这里就不体现操作步骤了
因为懒…所以就弄了一个master节点
以下的操作，只需要选一台可以和其他节点免密的 master 节点就好
网络条件好的情况下，镜像可以让他自己拉取，如果镜像经常拉取失败，可以从本地上传镜像包然后导入到 containerd，文章后面的镜像导入一类的操作不是必须要操作的
根据自身实际情况创建指定路径，此路径用来存放k8s二进制文件以及用到的镜像文件
mkdir -p /approot1/k8s/{bin,images,pkg,tmp/{ssl,service}}
关闭防火墙for i in 192.168.91.19 192.168.91.20;do ssh $i "systemctl disable firewalld --now"; done
关闭selinux临时关闭
for i in 192.168.91.19 192.168.91.20;do ssh $i "setenforce 0"; done
永久关闭
for i in 192.168.91.19 192.168.91.20;do ssh $i "sed -i '/SELINUX/s/enforcing/disabled/g' /etc/selinux/config"; done
关闭swap临时关闭
for i in 192.168.91.19 192.168.91.20;do ssh $i "swapoff -a"; done
永久关闭
for i in 192.168.91.19 192.168.91.20;do ssh $i "sed -i '/ swap / s/^(.*)$/#1/g' /etc/fstab"; done
开启内核模块临时开启
for i in 192.168.91.19 192.168.91.20;do ssh $i "modprobe ip_vs"; ssh $i "modprobe ip_vs_rr"; ssh $i "modprobe ip_vs_wrr"; ssh $i "modprobe ip_vs_sh"; ssh $i "modprobe nf_conntrack"; ssh $i "modprobe nf_conntrack_ipv4"; ssh $i "modprobe br_netfilter"; ssh $i "modprobe overlay"; done
永久开启
vim /approot1/k8s/tmp/service/k8s-modules.conf
ip_vsip_vs_rrip_vs_wrrip_vs_shnf_conntracknf_conntrack_ipv4br_netfilter
分发到所有节点for i in 192.168.91.19 192.168.91.20;do scp /approot1/k8s/tmp/service/k8s-modules.conf $i:/etc/modules-load.d/; done
启用systemd自动加载模块服务for i in 192.168.91.19 192.168.91.20;do ssh $i "systemctl enable systemd-modules-load"; ssh $i "systemctl restart systemd-modules-load"; ssh $i "systemctl is-active systemd-modules-load"; done
配置系统参数返回active表示 自动加载模块服务 启动成功
以下的参数适用于3.x和4.x系列的内核
vim /approot1/k8s/tmp/service/kubernetes.conf
建议编辑之前，在 vim 里面先执行 :set paste ，避免复制进去的内容和文档的不一致，比如多了注释，或者语法对齐异常
# 开启数据包转发功能（实现vxlan）net.ipv4.ip_forward=1# iptables对bridge的数据进行处理net.bridge.bridge-nf-call-iptables=1net.bridge.bridge-nf-call-ip6tables=1net.bridge.bridge-nf-call-arptables=1# 关闭tcp_tw_recycle，否则和NAT冲突，会导致服务不通net.ipv4.tcp_tw_recycle=0# 不允许将TIME-WAIT sockets重新用于新的TCP连接net.ipv4.tcp_tw_reuse=0# socket监听(listen)的backlog上限net.core.somaxconn=32768# 最大跟踪连接数，默认 nf_conntrack_buckets * 4net.netfilter.nf_conntrack_max=1000000# 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它vm.swappiness=0# 计算当前的内存映射文件数。vm.max_map_count=655360# 内核可分配的最大文件数fs.file-max=6553600# 持久连接net.ipv4.tcp_keepalive_time=600net.ipv4.tcp_keepalive_intvl=30net.ipv4.tcp_keepalive_probes=10
分发到所有节点for i in 192.168.91.19 192.168.91.20;do scp /approot1/k8s/tmp/service/kubernetes.conf $i:/etc/sysctl.d/; done
加载系统参数for i in 192.168.91.19 192.168.91.20;do ssh $i "sysctl -p /etc/sysctl.d/kubernetes.conf"; done
清空iptables规则for i in 192.168.91.19 192.168.91.20;do ssh $i "iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat"; ssh $i "iptables -P FORWARD ACCEPT"; done
配置 PATH 变量for i in 192.168.91.19 192.168.91.20;do ssh $i "echo 'PATH=$PATH:/approot1/k8s/bin' >> $HOME/.bashrc"; donesource $HOME/.bashrc
下载二进制文件其中一台节点操作即可
github下载会比较慢，可以从本地上传到 /approot1/k8s/pkg/ 目录下
wget -O /approot1/k8s/pkg/kubernetes.tar.gz https://dl.k8s.io/v1.23.3/kubernetes-server-linux-amd64.tar.gzwget -O /approot1/k8s/pkg/etcd.tar.gz https://github.com/etcd-io/etcd/releases/download/v3.5.1/etcd-v3.5.1-linux-amd64.tar.gz
解压并删除不必要的文件
cd /approot1/k8s/pkg/for i in $(ls *.tar.gz);do tar xvf $i && rm -f $i;donemv kubernetes/server/bin/ kubernetes/rm -rf kubernetes/{addons,kubernetes-src.tar.gz,LICENSES,server}rm -f kubernetes/bin/*_tag kubernetes/bin/*.tarrm -rf etcd-v3.5.1-linux-amd64/documentation etcd-v3.5.1-linux-amd64/*.md
部署 master 节点 创建 ca 根证书wget -O /approot1/k8s/bin/cfssl https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64wget -O /approot1/k8s/bin/cfssljson https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64chmod +x /approot1/k8s/bin/*
vim /approot1/k8s/tmp/ssl/ca-config.json
{ "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "876000h" } } }}
vim /approot1/k8s/tmp/ssl/ca-csr.json
{ "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "k8s", "OU": "System" } ], "ca": { "expiry": "876000h" }}
cd /approot1/k8s/tmp/ssl/cfssl gencert -initca ca-csr.json | cfssljson -bare ca
部署 etcd 组件 创建 etcd 证书vim /approot1/k8s/tmp/ssl/etcd-csr.json
这里的192.168.91.19需要改成自己的ip，不要一股脑的复制黏贴
注意json的格式
{ "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.91.19" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "k8s", "OU": "System" } ]}
cd /approot1/k8s/tmp/ssl/cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
配置 etcd 为 systemctl 管理vim /approot1/k8s/tmp/service/kube-etcd.service.192.168.91.19
这里的192.168.91.19需要改成自己的ip，不要一股脑的复制黏贴
etcd 参数
[Unit]Description=Etcd ServerAfter=network.targetAfter=network-online.targetWants=network-online.targetdocumentation=https://github.com/coreos[Service]Type=notifyWorkingDirectory=/approot1/k8s/data/etcdExecStart=/approot1/k8s/bin/etcd --name=etcd-192.168.91.19 --cert-file=/etc/kubernetes/ssl/etcd.pem --key-file=/etc/kubernetes/ssl/etcd-key.pem --peer-cert-file=/etc/kubernetes/ssl/etcd.pem --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem --trusted-ca-file=/etc/kubernetes/ssl/ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem --initial-advertise-peer-urls=https://192.168.91.19:2380 --listen-peer-urls=https://192.168.91.19:2380 --listen-client-urls=https://192.168.91.19:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.91.19:2379 --initial-cluster-token=etcd-cluster-0 --initial-cluster=etcd-192.168.91.19=https://192.168.91.19:2380 --initial-cluster-state=new --data-dir=/approot1/k8s/data/etcd --wal-dir= --snapshot-count=50000 --auto-compaction-retention=1 --auto-compaction-mode=periodic --max-request-bytes=10485760 --quota-backend-bytes=8589934592Restart=alwaysRestartSec=15LimitNOFILE=65536OOMScoreAdjust=-999[Install]WantedBy=multi-user.target
分发证书以及创建相关路径如果是多节点，只需要在192.168.91.19后面加上对应的ip即可，以空格为分隔，注意将192.168.91.19修改为自己的ip，切莫一股脑复制
对应的目录也要确保和自己规划的一致，如果和我的有不同，注意修改，否则服务会启动失败
for i in 192.168.91.19;do ssh $i "mkdir -p /etc/kubernetes/ssl"; ssh $i "mkdir -m 700 -p /approot1/k8s/data/etcd"; ssh $i "mkdir -p /approot1/k8s/bin"; scp /approot1/k8s/tmp/ssl/{ca*.pem,etcd*.pem} $i:/etc/kubernetes/ssl/; scp /approot1/k8s/tmp/service/kube-etcd.service.$i $i:/etc/systemd/system/kube-etcd.service; scp /approot1/k8s/pkg/etcd-v3.5.1-linux-amd64/etcd* $i:/approot1/k8s/bin/; done
启动 etcd 服务如果是多节点，只需要在192.168.91.19后面加上对应的ip即可，以空格为分隔，注意将192.168.91.19修改为自己的ip，切莫一股脑复制
for i in 192.168.91.19;do ssh $i "systemctl daemon-reload"; ssh $i "systemctl enable kube-etcd"; ssh $i "systemctl restart kube-etcd --no-block"; ssh $i "systemctl is-active kube-etcd"; done
返回 activating 表示 etcd 还在启动中，可以稍等一会，然后再执行 for i in 192.168.91.19;do ssh $i "systemctl is-active kube-etcd";done
返回active表示 etcd 启动成功，如果是多节点 etcd ，其中一个没有返回active属于正常的，可以使用下面的方式来验证集群
如果是多节点，只需要在192.168.91.19后面加上对应的ip即可，以空格为分隔，注意将192.168.91.19修改为自己的ip，切莫一股脑复制
for i in 192.168.91.19;do ssh $i "ETCDCTL_API=3 /approot1/k8s/bin/etcdctl --endpoints=https://${i}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem endpoint health"; done
部署 apiserver 组件 创建 apiserver 证书https://192.168.91.19:2379 is healthy: successfully committed proposal: took = 7.135668ms
返回以上信息，并显示 successfully 表示节点是健康的
vim /approot1/k8s/tmp/ssl/kubernetes-csr.json
这里的192.168.91.19需要改成自己的ip，不要一股脑的复制黏贴
注意json的格式
10.88.0.1 是 k8s 的服务 ip，千万不要和现有的网络一致，避免出现冲突
{ "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.91.19", "10.88.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "k8s", "OU": "System" } ]}
cd /approot1/k8s/tmp/ssl/cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
创建 metrics-server 证书vim /approot1/k8s/tmp/ssl/metrics-server-csr.json
{ "CN": "aggregator", "hosts": [ ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "k8s", "OU": "System" } ]}
cd /approot1/k8s/tmp/ssl/cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes metrics-server-csr.json | cfssljson -bare metrics-server
配置 apiserver 为 systemctl 管理vim /approot1/k8s/tmp/service/kube-apiserver.service.192.168.91.19
这里的192.168.91.19需要改成自己的ip，不要一股脑的复制黏贴
--service-cluster-ip-range 参数的 ip 网段要和 kubernetes-csr.json 里面的 10.88.0.1 是一个网段的
--etcd-servers 如果 etcd 是多节点的，这里要写上所有的 etcd 节点
apiserver 参数
[Unit]Description=Kubernetes API Serverdocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=network.target[Service]ExecStart=/approot1/k8s/bin/kube-apiserver --allow-privileged=true --anonymous-auth=false --api-audiences=api,istio-ca --authorization-mode=Node,RBAC --bind-address=192.168.91.19 --client-ca-file=/etc/kubernetes/ssl/ca.pem --endpoint-reconciler-type=lease --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --etcd-servers=https://192.168.91.19:2379 --kubelet-certificate-authority=/etc/kubernetes/ssl/ca.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kubernetes.pem --kubelet-client-key=/etc/kubernetes/ssl/kubernetes-key.pem --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem --service-account-key-file=/etc/kubernetes/ssl/ca.pem --service-cluster-ip-range=10.88.0.0/16 --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem --requestheader-allowed-names= --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --proxy-client-cert-file=/etc/kubernetes/ssl/metrics-server.pem --proxy-client-key-file=/etc/kubernetes/ssl/metrics-server-key.pem --enable-aggregator-routing=true --v=2Restart=alwaysRestartSec=5Type=notifyLimitNOFILE=65536[Install]WantedBy=multi-user.target
分发证书以及创建相关路径如果是多节点，只需要在192.168.91.19后面加上对应的ip即可，以空格为分隔，注意将192.168.91.19修改为自己的ip，切莫一股脑复制
对应的目录也要确保和自己规划的一致，如果和我的有不同，注意修改，否则服务会启动失败
for i in 192.168.91.19;do ssh $i "mkdir -p /etc/kubernetes/ssl"; ssh $i "mkdir -p /approot1/k8s/bin"; scp /approot1/k8s/tmp/ssl/{ca*.pem,kubernetes*.pem,metrics-server*.pem} $i:/etc/kubernetes/ssl/; scp /approot1/k8s/tmp/service/kube-apiserver.service.$i $i:/etc/systemd/system/kube-apiserver.service; scp /approot1/k8s/pkg/kubernetes/bin/kube-apiserver $i:/approot1/k8s/bin/; done
启动 apiserver 服务如果是多节点，只需要在192.168.91.19后面加上对应的ip即可，以空格为分隔，注意将192.168.91.19修改为自己的ip，切莫一股脑复制
for i in 192.168.91.19;do ssh $i "systemctl daemon-reload"; ssh $i "systemctl enable kube-apiserver"; ssh $i "systemctl restart kube-apiserver --no-block"; ssh $i "systemctl is-active kube-apiserver"; done
返回 activating 表示 apiserver 还在启动中，可以稍等一会，然后再执行 for i in 192.168.91.19;do ssh $i "systemctl is-active kube-apiserver";done
返回active表示 apiserver 启动成功
curl -k --cacert /etc/kubernetes/ssl/ca.pem --cert /etc/kubernetes/ssl/kubernetes.pem --key /etc/kubernetes/ssl/kubernetes-key.pem https://192.168.91.19:6443/api
正常返回如下信息，说明 apiserver 服务运行正常
{ "kind": "APIVersions", "versions": [ "v1" ], "serverAddressByClientCIDRs": [ { "clientCIDR": "0.0.0.0/0", "serverAddress": "192.168.91.19:6443" } ]}
查看 k8s 的所有 kind （对象类别）
curl -s -k --cacert /etc/kubernetes/ssl/ca.pem --cert /etc/kubernetes/ssl/kubernetes.pem --key /etc/kubernetes/ssl/kubernetes-key.pem https://192.168.91.19:6443/api/v1/ | grep kind | sort -u
"kind": "APIResourceList", "kind": "Binding", "kind": "ComponentStatus", "kind": "ConfigMap", "kind": "Endpoints", "kind": "Event", "kind": "Eviction", "kind": "LimitRange", "kind": "Namespace", "kind": "Node", "kind": "NodeProxyOptions", "kind": "PersistentVolume", "kind": "PersistentVolumeClaim", "kind": "Pod", "kind": "PodAttachOptions", "kind": "PodExecOptions", "kind": "PodPortForwardOptions", "kind": "PodProxyOptions", "kind": "PodTemplate", "kind": "ReplicationController", "kind": "ResourceQuota", "kind": "Scale", "kind": "Secret", "kind": "Service", "kind": "ServiceAccount", "kind": "ServiceProxyOptions", "kind": "TokenRequest",
配置 kubectl 管理 创建 admin 证书vim /approot1/k8s/tmp/ssl/admin-csr.json
{ "CN": "admin", "hosts": [ ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "system:masters", "OU": "System" } ]}
cd /approot1/k8s/tmp/ssl/cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
创建 kubeconfig 证书设置集群参数
--server 为 apiserver 的访问地址，修改成自己的 ip 地址和 service 文件里面指定的 --secure-port 参数的端口，切记，一定要带上https:// 协议，否则生成的证书，kubectl 命令访问不到 apiserver
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.91.19:6443 --kubeconfig=kubectl.kubeconfig
设置客户端认证参数
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=kubectl.kubeconfig
设置上下文参数
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=kubectl.kubeconfig
设置默认上下文
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config use-context kubernetes --kubeconfig=kubectl.kubeconfig
分发 kubeconfig 证书到所有 master 节点如果是多节点，只需要在192.168.91.19后面加上对应的ip即可，以空格为分隔，注意将192.168.91.19修改为自己的ip，切莫一股脑复制
for i in 192.168.91.19;do ssh $i "mkdir -p /etc/kubernetes/ssl"; ssh $i "mkdir -p /approot1/k8s/bin"; ssh $i "mkdir -p $HOME/.kube"; ssh $i "echo 'source <(kubectl completion bash)' >> $HOME/.bashrc"scp /approot1/k8s/tmp/ssl/kubectl.kubeconfig $i:$HOME/.kube/config; scp /approot1/k8s/pkg/kubernetes/bin/kubectl $i:/approot1/k8s/bin/; done
部署 controller-manager 组件 创建 controller-manager 证书vim /approot1/k8s/tmp/ssl/kube-controller-manager-csr.json
这里的192.168.91.19需要改成自己的ip，不要一股脑的复制黏贴
注意json的格式
{ "CN": "system:kube-controller-manager", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.91.19" ], "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "system:kube-controller-manager", "OU": "System" } ]}
cd /approot1/k8s/tmp/ssl/cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
创建 kubeconfig 证书设置集群参数
--server 为 apiserver 的访问地址，修改成自己的 ip 地址和 service 文件里面指定的 --secure-port 参数的端口，切记，一定要带上https:// 协议，否则生成的证书，kubectl 命令访问不到 apiserver
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.91.19:6443 --kubeconfig=kube-controller-manager.kubeconfig
设置客户端认证参数
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig
设置上下文参数
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
设置默认上下文
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
配置 controller-manager 为 systemctl 管理vim /approot1/k8s/tmp/service/kube-controller-manager.service
这里的192.168.91.19需要改成自己的ip，不要一股脑的复制黏贴
--service-cluster-ip-range 参数的 ip 网段要和 kubernetes-csr.json 里面的 10.88.0.1 是一个网段的
--cluster-cidr 为 pod 运行的网段，要和 --service-cluster-ip-range 参数的网段以及现有的网络不一致，避免出现冲突
controller-manager 参数
[Unit]Description=Kubernetes Controller Managerdocumentation=https://github.com/GoogleCloudPlatform/kubernetes[Service]ExecStart=/approot1/k8s/bin/kube-controller-manager --bind-address=0.0.0.0 --allocate-node-cidrs=true --cluster-cidr=172.20.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig --leader-elect=true --node-cidr-mask-size=24 --root-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem --service-cluster-ip-range=10.88.0.0/16 --use-service-account-credentials=true --v=2Restart=alwaysRestartSec=5[Install]WantedBy=multi-user.target
分发证书以及创建相关路径如果是多节点，只需要在192.168.91.19后面加上对应的ip即可，以空格为分隔，注意将192.168.91.19修改为自己的ip，切莫一股脑复制
对应的目录也要确保和自己规划的一致，如果和我的有不同，注意修改，否则服务会启动失败
for i in 192.168.91.19;do ssh $i "mkdir -p /etc/kubernetes/ssl"; ssh $i "mkdir -p /approot1/k8s/bin"; scp /approot1/k8s/tmp/ssl/kube-controller-manager.kubeconfig $i:/etc/kubernetes/; scp /approot1/k8s/tmp/ssl/ca*.pem $i:/etc/kubernetes/ssl/; scp /approot1/k8s/tmp/service/kube-controller-manager.service $i:/etc/systemd/system/; scp /approot1/k8s/pkg/kubernetes/bin/kube-controller-manager $i:/approot1/k8s/bin/; done
启动 controller-manager 服务如果是多节点，只需要在192.168.91.19后面加上对应的ip即可，以空格为分隔，注意将192.168.91.19修改为自己的ip，切莫一股脑复制
for i in 192.168.91.19;do ssh $i "systemctl daemon-reload"; ssh $i "systemctl enable kube-controller-manager"; ssh $i "systemctl restart kube-controller-manager --no-block"; ssh $i "systemctl is-active kube-controller-manager"; done
部署 scheduler 组件 创建 scheduler 证书返回 activating 表示 controller-manager 还在启动中，可以稍等一会，然后再执行 for i in 192.168.91.19;do ssh $i "systemctl is-active kube-controller-manager";done
返回active表示 controller-manager 启动成功
vim /approot1/k8s/tmp/ssl/kube-scheduler-csr.json
这里的192.168.91.19需要改成自己的ip，不要一股脑的复制黏贴
注意json的格式
{ "CN": "system:kube-scheduler", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.91.19" ], "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "system:kube-scheduler", "OU": "System" } ]}
cd /approot1/k8s/tmp/ssl/cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
创建 kubeconfig 证书设置集群参数
--server 为 apiserver 的访问地址，修改成自己的 ip 地址和 service 文件里面指定的 --secure-port 参数的端口，切记，一定要带上https:// 协议，否则生成的证书，kubectl 命令访问不到 apiserver
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.91.19:6443 --kubeconfig=kube-scheduler.kubeconfig
设置客户端认证参数
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig
设置上下文参数
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
设置默认上下文
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
配置 scheduler 为 systemctl 管理vim /approot1/k8s/tmp/service/kube-scheduler.service
scheduler 参数
[Unit]Description=Kubernetes Schedulerdocumentation=https://github.com/GoogleCloudPlatform/kubernetes[Service]ExecStart=/approot1/k8s/bin/kube-scheduler --authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig --authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig --bind-address=0.0.0.0 --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig --leader-elect=true --v=2Restart=alwaysRestartSec=5[Install]WantedBy=multi-user.target
分发证书以及创建相关路径如果是多节点，只需要在192.168.91.19后面加上对应的ip即可，以空格为分隔，注意将192.168.91.19修改为自己的ip，切莫一股脑复制
对应的目录也要确保和自己规划的一致，如果和我的有不同，注意修改，否则服务会启动失败
for i in 192.168.91.19;do ssh $i "mkdir -p /etc/kubernetes/ssl"; ssh $i "mkdir -p /approot1/k8s/bin"; scp /approot1/k8s/tmp/ssl/{ca*.pem,kube-scheduler.kubeconfig} $i:/etc/kubernetes/; scp /approot1/k8s/tmp/service/kube-scheduler.service $i:/etc/systemd/system/; scp /approot1/k8s/pkg/kubernetes/bin/kube-scheduler $i:/approot1/k8s/bin/; done
启动 scheduler 服务如果是多节点，只需要在192.168.91.19后面加上对应的ip即可，以空格为分隔，注意将192.168.91.19修改为自己的ip，切莫一股脑复制
for i in 192.168.91.19;do ssh $i "systemctl daemon-reload"; ssh $i "systemctl enable kube-scheduler"; ssh $i "systemctl restart kube-scheduler --no-block"; ssh $i "systemctl is-active kube-scheduler"; done
部署 work 节点 部署 containerd 组件 下载二进制文件返回 activating 表示 scheduler 还在启动中，可以稍等一会，然后再执行 for i in 192.168.91.19;do ssh $i "systemctl is-active kube-scheduler";done
返回active表示 scheduler 启动成功
github 下载 containerd 的时候，记得选择cri-containerd-cni 开头的文件，这个包里面包含了 containerd 以及 crictl 管理工具和 cni 网络插件，包括 systemd service 文件、config.toml 、 crictl.yaml 以及 cni 配置文件都是配置好的，简单修改一下就可以使用了
虽然 cri-containerd-cni 也有 runc ，但是缺少依赖，所以还是要去 runc github 重新下载一个
wget -O /approot1/k8s/pkg/containerd.tar.gz https://github.com/containerd/containerd/releases/download/v1.5.9/cri-containerd-cni-1.5.9-linux-amd64.tar.gzwget -O /approot1/k8s/pkg/runc https://github.com/opencontainers/runc/releases/download/v1.0.3/runc.amd64mkdir /approot1/k8s/pkg/containerdcd /approot1/k8s/pkg/for i in $(ls *containerd*.tar.gz);do tar xvf $i -C /approot1/k8s/pkg/containerd && rm -f $i;donechmod +x /approot1/k8s/pkg/runcmv /approot1/k8s/pkg/containerd/usr/local/bin/{containerd,containerd-shim*,crictl,ctr} /approot1/k8s/pkg/containerd/mv /approot1/k8s/pkg/containerd/opt/cni/bin/{bridge,flannel,host-local,loopback,portmap} /approot1/k8s/pkg/containerd/rm -rf /approot1/k8s/pkg/containerd/{etc,opt,usr}
配置 containerd 为 systemctl 管理vim /approot1/k8s/tmp/service/containerd.service
注意二进制文件存放路径
如果 runc 二进制文件不在 /usr/bin/ 目录下，需要有 Environment 参数，指定 runc 二进制文件的路径给 PATH ，否则当 k8s 启动 pod 的时候会报错 exec: "runc": executable file not found in $PATH: unknown
[Unit]Description=containerd container runtimedocumentation=https://containerd.ioAfter=network.target[Service]Environment="PATH=$PATH:/approot1/k8s/bin"ExecStartPre=-/sbin/modprobe overlayExecStart=/approot1/k8s/bin/containerdRestart=alwaysRestartSec=5Delegate=yesKillMode=processOOMScoreAdjust=-999LimitNOFILE=1048576# Having non-zero Limit*s causes performance problems due to accounting overhead# in the kernel、We recommend using cgroups to do container-local accounting.LimitNPROC=infinityLimitCORE=infinity[Install]WantedBy=multi-user.target
配置 containerd 配置文件vim /approot1/k8s/tmp/service/config.toml
root 容器存储路径，修改成磁盘空间充足的路径
bin_dir containerd 服务以及 cni 插件存储路径
sandbox_image pause 镜像名称以及镜像tag
disabled_plugins = []imports = []oom_score = 0plugin_dir = ""required_plugins = []root = "/approot1/data/containerd"state = "/run/containerd"version = 2[cgroup] path = ""[debug] address = "" format = "" gid = 0 level = "" uid = 0[grpc] address = "/run/containerd/containerd.sock" gid = 0 max_recv_message_size = 16777216 max_send_message_size = 16777216 tcp_address = "" tcp_tls_cert = "" tcp_tls_key = "" uid = 0[metrics] address = "" grpc_histogram = false[plugins] [plugins."io.containerd.gc.v1.scheduler"] deletion_threshold = 0 mutation_threshold = 100 pause_threshold = 0.02 schedule_delay = "0s" startup_delay = "100ms" [plugins."io.containerd.grpc.v1.cri"] disable_apparmor = false disable_cgroup = false disable_hugetlb_controller = true disable_proc_mount = false disable_tcp_service = true enable_selinux = false enable_tls_streaming = false ignore_image_defined_volumes = false max_concurrent_downloads = 3 max_container_log_line_size = 16384 netns_mounts_under_state_dir = false restrict_oom_score_adj = false sandbox_image = "k8s.gcr.io/pause:3.6" selinux_category_range = 1024 stats_collect_period = 10 stream_idle_timeout = "4h0m0s" stream_server_address = "127.0.0.1" stream_server_port = "0" systemd_cgroup = false tolerate_missing_hugetlb_controller = true unset_seccomp_profile = "" [plugins."io.containerd.grpc.v1.cri".cni] bin_dir = "/approot1/k8s/bin" conf_dir = "/etc/cni/net.d" conf_template = "/etc/cni/net.d/cni-default.conf" max_conf_num = 1 [plugins."io.containerd.grpc.v1.cri".containerd] default_runtime_name = "runc" disable_snapshot_annotations = true discard_unpacked_layers = false no_pivot = false snapshotter = "overlayfs" [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime] base_runtime_spec = "" container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_root = "" runtime_type = "" [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] base_runtime_spec = "" container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_root = "" runtime_type = "io.containerd.runc.v2" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] BinaryName = "" CriuImagePath = "" CriuPath = "" CriuWorkPath = "" IoGid = 0 IoUid = 0 NonewKeyring = false NoPivotRoot = false Root = "" ShimCgroup = "" SystemdCgroup = true [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime] base_runtime_spec = "" container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_root = "" runtime_type = "" [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options] [plugins."io.containerd.grpc.v1.cri".image_decryption] key_model = "node" [plugins."io.containerd.grpc.v1.cri".registry] config_path = "" [plugins."io.containerd.grpc.v1.cri".registry.auths] [plugins."io.containerd.grpc.v1.cri".registry.configs] [plugins."io.containerd.grpc.v1.cri".registry.headers] [plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = ["https://docker.mirrors.ustc.edu.cn", "http://hub-mirror.c.163.com"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"] endpoint = ["https://gcr.mirrors.ustc.edu.cn"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"] endpoint = ["https://gcr.mirrors.ustc.edu.cn/google-containers/"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"] endpoint = ["https://quay.mirrors.ustc.edu.cn"] [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming] tls_cert_file = "" tls_key_file = "" [plugins."io.containerd.internal.v1.opt"] path = "/opt/containerd" [plugins."io.containerd.internal.v1.restart"] interval = "10s" [plugins."io.containerd.metadata.v1.bolt"] content_sharing_policy = "shared" [plugins."io.containerd.monitor.v1.cgroups"] no_prometheus = false [plugins."io.containerd.runtime.v1.linux"] no_shim = false runtime = "runc" runtime_root = "" shim = "containerd-shim" shim_debug = false [plugins."io.containerd.runtime.v2.task"] platforms = ["linux/amd64"] [plugins."io.containerd.service.v1.diff-service"] default = ["walking"] [plugins."io.containerd.snapshotter.v1.aufs"] root_path = "" [plugins."io.containerd.snapshotter.v1.btrfs"] root_path = "" [plugins."io.containerd.snapshotter.v1.devmapper"] async_remove = false base_image_size = "" pool_name = "" root_path = "" [plugins."io.containerd.snapshotter.v1.native"] root_path = "" [plugins."io.containerd.snapshotter.v1.overlayfs"] root_path = "" [plugins."io.containerd.snapshotter.v1.zfs"] root_path = ""[proxy_plugins][stream_processors] [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"] accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"] args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"] env = ["OCICRYPT_KEYPROVIDER_ConFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"] path = "ctd-decoder" returns = "application/vnd.oci.image.layer.v1.tar" [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"] accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"] args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"] env = ["OCICRYPT_KEYPROVIDER_ConFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"] path = "ctd-decoder" returns = "application/vnd.oci.image.layer.v1.tar+gzip"[timeouts] "io.containerd.timeout.shim.cleanup" = "5s" "io.containerd.timeout.shim.load" = "5s" "io.containerd.timeout.shim.shutdown" = "3s" "io.containerd.timeout.task.state" = "2s"[ttrpc] address = "" gid = 0 uid = 0
配置 crictl 管理工具vim /approot1/k8s/tmp/service/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
配置 cni 网络插件vim /approot1/k8s/tmp/service/cni-default.conf
subnet 参数要和 controller-manager 的 --cluster-cidr 参数一致
{"name": "mynet","cniVersion": "0.3.1","type": "bridge","bridge": "mynet0","isDefaultGateway": true,"ipMasq": true,"hairpinMode": true,"ipam": {"type": "host-local","subnet": "172.20.0.0/16"}}
分发配置文件以及创建相关路径for i in 192.168.91.19 192.168.91.20;do ssh $i "mkdir -p /etc/containerd"; ssh $i "mkdir -p /approot1/k8s/bin"; ssh $i "mkdir -p /etc/cni/net.d"; scp /approot1/k8s/tmp/service/containerd.service $i:/etc/systemd/system/; scp /approot1/k8s/tmp/service/config.toml $i:/etc/containerd/; scp /approot1/k8s/tmp/service/cni-default.conf $i:/etc/cni/net.d/; scp /approot1/k8s/tmp/service/crictl.yaml $i:/etc/; scp /approot1/k8s/pkg/containerd/* $i:/approot1/k8s/bin/; scp /approot1/k8s/pkg/runc $i:/approot1/k8s/bin/; done
启动 containerd 服务for i in 192.168.91.19 192.168.91.20;do ssh $i "systemctl daemon-reload"; ssh $i "systemctl enable containerd"; ssh $i "systemctl restart containerd --no-block"; ssh $i "systemctl is-active containerd"; done
导入 pause 镜像返回 activating 表示 containerd 还在启动中，可以稍等一会，然后再执行 for i in 192.168.91.19 192.168.91.20;do ssh $i "systemctl is-active containerd";done
返回active表示 containerd 启动成功
ctr 导入镜像有一个特殊的地方，如果导入的镜像想要 k8s 可以使用，需要加上 -n k8s.io 参数，而且必须是ctr -n k8s.io image import
这样的格式，如果是 ctr image import -n k8s.io 就会报错 ctr: flag provided but not defined: -n 这个操作确实有点骚气，不太适应 如果镜像导入的时候没有加上 -n k8s.io ，启动 pod 的时候 kubelet 会重新去拉取 pause 容器，如果配置的镜像仓库没有这个 tag 的镜像就会报错
for i in 192.168.91.19 192.168.91.20;do scp /approot1/k8s/images/pause-v3.6.tar $i:/tmp/ssh $i "ctr -n=k8s.io image import /tmp/pause-v3.6.tar && rm -f /tmp/pause-v3.6.tar"; done
查看镜像
for i in 192.168.91.19 192.168.91.20;do ssh $i "ctr -n=k8s.io image list | grep pause"; done
部署 kubelet 组件 创建 kubelet 证书vim /approot1/k8s/tmp/ssl/kubelet-csr.json.192.168.91.19
这里的192.168.91.19需要改成自己的ip，不要一股脑的复制黏贴，有多少个node节点就创建多少个json文件，json文件内的 ip 也要修改为 work 节点的 ip，别重复了
{ "CN": "system:node:192.168.91.19", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.91.19" ], "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "system:nodes", "OU": "System" } ]}
for i in 192.168.91.19 192.168.91.20;do cd /approot1/k8s/tmp/ssl/; cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubelet-csr.json.$i | cfssljson -bare kubelet.$i; done
创建 kubeconfig 证书设置集群参数
--server 为 apiserver 的访问地址，修改成自己的 ip 地址和 service 文件里面指定的 --secure-port 参数的端口，切记，一定要带上https:// 协议，否则生成的证书，kubectl 命令访问不到 apiserver
for i in 192.168.91.19 192.168.91.20;do cd /approot1/k8s/tmp/ssl/; /approot1/k8s/pkg/kubernetes/bin/kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.91.19:6443 --kubeconfig=kubelet.kubeconfig.$i; done
设置客户端认证参数
for i in 192.168.91.19 192.168.91.20;do cd /approot1/k8s/tmp/ssl/; /approot1/k8s/pkg/kubernetes/bin/kubectl config set-credentials system:node:$i --client-certificate=kubelet.$i.pem --client-key=kubelet.$i-key.pem --embed-certs=true --kubeconfig=kubelet.kubeconfig.$i; done
设置上下文参数
for i in 192.168.91.19 192.168.91.20;do cd /approot1/k8s/tmp/ssl/; /approot1/k8s/pkg/kubernetes/bin/kubectl config set-context default --cluster=kubernetes --user=system:node:$i --kubeconfig=kubelet.kubeconfig.$i; done
设置默认上下文
for i in 192.168.91.19 192.168.91.20;do cd /approot1/k8s/tmp/ssl/; /approot1/k8s/pkg/kubernetes/bin/kubectl config use-context default --kubeconfig=kubelet.kubeconfig.$i; done
配置 kubelet 配置文件vim /approot1/k8s/tmp/service/config.yaml
clusterDNS 参数的 ip 注意修改，和 apiserver 的 --service-cluster-ip-range 参数一个网段，和 k8s 服务 ip 要不一样，一般 k8s 服务的 ip 取网段第一个ip， clusterdns 选网段的第二个ip
kind: KubeletConfigurationapiVersion: kubelet.config.k8s.io/v1beta1address: 0.0.0.0authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /etc/kubernetes/ssl/ca.pemauthorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30scgroupDriver: systemdcgroupsPerQOS: trueclusterDNS:- 10.88.0.2clusterDomain: cluster.localconfigMapAndSecretChangeDetectionStrategy: WatchcontainerLogMaxFiles: 3containerLogMaxSize: 10MienforceNodeAllocatable:- podseventBurst: 10eventRecordQPS: 5evictionHard: imagefs.available: 15% memory.available: 300Mi nodefs.available: 10% nodefs.inodesFree: 5%evictionPressureTransitionPeriod: 5m0sfailSwapOn: truefileCheckFrequency: 40shairpinMode: hairpin-vethhealthzBindAddress: 0.0.0.0healthzPort: 10248httpCheckFrequency: 40simageGCHighThresholdPercent: 85imageGCLowThresholdPercent: 80imageMinimumGCAge: 2m0skubeAPIBurst: 100kubeAPIQPS: 50makeIPTablesUtilChains: truemaxOpenFiles: 1000000maxPods: 110nodeLeaseDurationSeconds: 40nodeStatusReportFrequency: 1m0snodeStatusUpdateFrequency: 10soomScoreAdj: -999podPidsLimit: -1port: 10250# disable readOnlyPortreadOnlyPort: 0resolvConf: /etc/resolv.confruntimeRequestTimeout: 2m0sserializeImagePulls: truestreamingConnectionIdleTimeout: 4h0m0ssyncFrequency: 1m0stlsCertFile: /etc/kubernetes/ssl/kubelet.pemtlsPrivateKeyFile: /etc/kubernetes/ssl/kubelet-key.pem
配置 kubelet 为 systemctl 管理vim /approot1/k8s/tmp/service/kubelet.service.192.168.91.19
这里的192.168.91.19需要改成自己的ip，不要一股脑的复制黏贴，有多少个node节点就创建多少个service文件，service 文件内的 ip 也要修改为 work 节点的 ip，别重复了
--container-runtime 参数默认是 docker ，如果使用 docker 以外的，需要配置为 remote ，并且要配置 --container-runtime-endpoint 参数来指定 sock 文件的路径
kubelet 参数
[Unit]Description=Kubernetes Kubeletdocumentation=https://github.com/GoogleCloudPlatform/kubernetes[Service]WorkingDirectory=/approot1/k8s/data/kubeletExecStart=/approot1/k8s/bin/kubelet --config=/approot1/k8s/data/kubelet/config.yaml --cni-bin-dir=/approot1/k8s/bin --cni-conf-dir=/etc/cni/net.d --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --hostname-override=192.168.91.19 --image-pull-progress-deadline=5m --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.6 --root-dir=/approot1/k8s/data/kubelet --v=2Restart=alwaysRestartSec=5[Install]WantedBy=multi-user.target
分发证书以及创建相关路径如果是多节点，只需要在192.168.91.19后面加上对应的ip即可，以空格为分隔，注意将192.168.91.19修改为自己的ip，切莫一股脑复制
对应的目录也要确保和自己规划的一致，如果和我的有不同，注意修改，否则服务会启动失败
for i in 192.168.91.19 192.168.91.20;do ssh $i "mkdir -p /approot1/k8s/data/kubelet"; ssh $i "mkdir -p /approot1/k8s/bin"; ssh $i "mkdir -p /etc/kubernetes/ssl"; scp /approot1/k8s/tmp/ssl/ca*.pem $i:/etc/kubernetes/ssl/; scp /approot1/k8s/tmp/ssl/kubelet.$i.pem $i:/etc/kubernetes/ssl/kubelet.pem; scp /approot1/k8s/tmp/ssl/kubelet.$i-key.pem $i:/etc/kubernetes/ssl/kubelet-key.pem; scp /approot1/k8s/tmp/ssl/kubelet.kubeconfig.$i $i:/etc/kubernetes/kubelet.kubeconfig; scp /approot1/k8s/tmp/service/kubelet.service.$i $i:/etc/systemd/system/kubelet.service; scp /approot1/k8s/tmp/service/config.yaml $i:/approot1/k8s/data/kubelet/; scp /approot1/k8s/pkg/kubernetes/bin/kubelet $i:/approot1/k8s/bin/; done
启动 kubelet 服务for i in 192.168.91.19 192.168.91.20;do ssh $i "systemctl daemon-reload"; ssh $i "systemctl enable kubelet"; ssh $i "systemctl restart kubelet --no-block"; ssh $i "systemctl is-active kubelet"; done
查看节点是否 Ready返回 activating 表示 kubelet 还在启动中，可以稍等一会，然后再执行 for i in 192.168.91.19 192.168.91.20;do ssh $i "systemctl is-active kubelet";done
返回active表示 kubelet 启动成功
kubectl get node
预期出现类似如下输出，STATUS 字段为 Ready 表示节点正常
NAME STATUS ROLES AGE VERSION192.168.91.19 Ready
vim /approot1/k8s/tmp/ssl/kube-proxy-csr.json
{ "CN": "system:kube-proxy", "key": { "algo": "rsa", "size": 2048 }, "hosts": [], "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai", "O": "system:kube-proxy", "OU": "System" } ]}
cd /approot1/k8s/tmp/ssl/; cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
创建 kubeconfig 证书设置集群参数
--server 为 apiserver 的访问地址，修改成自己的 ip 地址和 service 文件里面指定的 --secure-port 参数的端口，切记，一定要带上https:// 协议，否则生成的证书，kubectl 命令访问不到 apiserver
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.91.19:6443 --kubeconfig=kube-proxy.kubeconfig
设置客户端认证参数
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig
设置上下文参数
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig
设置默认上下文
cd /approot1/k8s/tmp/ssl//approot1/k8s/pkg/kubernetes/bin/kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
配置 kube-proxy 配置文件vim /approot1/k8s/tmp/service/kube-proxy-config.yaml.192.168.91.19
这里的192.168.91.19需要改成自己的ip，不要一股脑的复制黏贴，有多少个node节点就创建多少个service文件，service 文件内的 ip 也要修改为 work 节点的 ip，别重复了
clusterCIDR 参数要和 controller-manager 的 --cluster-cidr 参数一致
hostnameOverride 要和 kubelet 的 --hostname-override 参数一致，否则会出现 node not found 的报错
kind: KubeProxyConfigurationapiVersion: kubeproxy.config.k8s.io/v1alpha1bindAddress: 0.0.0.0clientConnection: kubeconfig: "/etc/kubernetes/kube-proxy.kubeconfig"clusterCIDR: "172.20.0.0/16"conntrack: maxPerCore: 32768 min: 131072 tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0shealthzBindAddress: 0.0.0.0:10256hostnameOverride: "192.168.91.19"metricsBindAddress: 0.0.0.0:10249mode: "ipvs"
配置 proxy 为 systemctl 管理vim /approot1/k8s/tmp/service/kube-proxy.service
[Unit]Description=Kubernetes Kube-Proxy Serverdocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=network.target[Service]# kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量## 指定 --cluster-cidr 或 --masquerade-all 选项后## kube-proxy 会对访问 Service IP 的请求做 SNATWorkingDirectory=/approot1/k8s/data/kube-proxyExecStart=/approot1/k8s/bin/kube-proxy --config=/approot1/k8s/data/kube-proxy/kube-proxy-config.yamlRestart=alwaysRestartSec=5LimitNOFILE=65536[Install]WantedBy=multi-user.target
分发证书以及创建相关路径如果是多节点，只需要在192.168.91.19后面加上对应的ip即可，以空格为分隔，注意将192.168.91.19修改为自己的ip，切莫一股脑复制
对应的目录也要确保和自己规划的一致，如果和我的有不同，注意修改，否则服务会启动失败
for i in 192.168.91.19 192.168.91.20;do ssh $i "mkdir -p /approot1/k8s/data//kube-proxy"; ssh $i "mkdir -p /approot1/k8s/bin"; ssh $i "mkdir -p /etc/kubernetes/ssl"; scp /approot1/k8s/tmp/ssl/kube-proxy.kubeconfig $i:/etc/kubernetes/; scp /approot1/k8s/tmp/service/kube-proxy.service $i:/etc/systemd/system/; scp /approot1/k8s/tmp/service/kube-proxy-config.yaml.$i $i:/approot1/k8s/data/kube-proxy/kube-proxy-config.yaml; scp /approot1/k8s/pkg/kubernetes/bin/kube-proxy $i:/approot1/k8s/bin/; done
启动 kube-proxy 服务for i in 192.168.91.19 192.168.91.20;do ssh $i "systemctl daemon-reload"; ssh $i "systemctl enable kube-proxy"; ssh $i "systemctl restart kube-proxy --no-block"; ssh $i "systemctl is-active kube-proxy"; done
部署 flannel 组件返回 activating 表示 kubelet 还在启动中，可以稍等一会，然后再执行 for i in 192.168.91.19 192.168.91.20;do ssh $i "systemctl is-active kubelet";done
返回active表示 kubelet 启动成功
配置 flannel yaml 文件flannel github
vim /approot1/k8s/tmp/service/flannel.yaml
net-conf.json 内的 Network 参数需要和 controller-manager 的 --cluster-cidr 参数一致
---apiVersion: policy/v1beta1kind: PodSecurityPolicymetadata: name: psp.flannel.unprivileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/defaultspec: privileged: false volumes: - configMap - secret - emptyDir - hostPath allowedHostPaths: - pathPrefix: "/etc/cni/net.d" - pathPrefix: "/etc/kube-flannel" - pathPrefix: "/run/flannel" readOnlyRootFilesystem: false # Users and groups runAsUser: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny # Privilege Escalation allowPrivilegeEscalation: false defaultAllowPrivilegeEscalation: false # Capabilities allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] defaultAddCapabilities: [] requiredDropCapabilities: [] # Host namespaces hostPID: false hostIPC: false hostNetwork: true hostPorts: - min: 0 max: 65535 # SELinux seLinux: # SELinux is unused in CaaSP rule: 'RunAsAny'---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata: name: flannelrules:- apiGroups: ['policy'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: ['psp.flannel.unprivileged']- apiGroups: - "" resources: - pods verbs: - get- apiGroups: - "" resources: - nodes verbs: - list - watch- apiGroups: - "" resources: - nodes/status verbs: - patch---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata: name: flannelroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannelsubjects:- kind: ServiceAccount name: flannel namespace: kube-system---apiVersion: v1kind: ServiceAccountmetadata: name: flannel namespace: kube-system---kind: ConfigMapapiVersion: v1metadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node app: flanneldata: cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } net-conf.json: | { "Network": "172.20.0.0/16", "Backend": { "Type": "vxlan" } }---apiVersion: apps/v1kind: DaemonSetmetadata: name: kube-flannel-ds namespace: kube-system labels: tier: node app: flannelspec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchexpressions: - key: kubernetes.io/os operator: In values: - linux hostNetwork: true priorityClassName: system-node-critical tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay.io/coreos/flannel:v0.15.1 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.15.1 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN", "NET_RAW"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg
配置 flannel cni 网卡配置文件vim /approot1/k8s/tmp/service/10-flannel.conflist
{ "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ]}
导入 flannel 镜像for i in 192.168.91.19 192.168.91.20;do scp /approot1/k8s/images/flannel-v0.15.1.tar $i:/tmp/ssh $i "ctr -n=k8s.io image import /tmp/flannel-v0.15.1.tar && rm -f /tmp/flannel-v0.15.1.tar"; done
查看镜像
for i in 192.168.91.19 192.168.91.20;do ssh $i "ctr -n=k8s.io image list | grep flannel"; done
分发 flannel cni 网卡配置文件for i in 192.168.91.19 192.168.91.20;do ssh $i "rm -f /etc/cni/net.d/10-default.conf"; scp /approot1/k8s/tmp/service/10-flannel.conflist $i:/etc/cni/net.d/; done
在 k8s 中运行 flannel 组件kubectl apply -f /approot1/k8s/tmp/service/flannel.yaml
检查 flannel pod 是否运行成功kubectl get pod -n kube-system | grep flannel
预期输出类似如下结果
flannel 属于 DaemonSet ，属于和节点共存亡类型的 pod ，k8s 有多少 node ，flannel 就有多少 pod ，当 node 被删除的时候， flannel pod 也会随之删除
kube-flannel-ds-86rrv 1/1 Running 0 8m54skube-flannel-ds-bkgzx 1/1 Running 0 8m53s
部署 coredns 组件 配置 coredns yaml 文件vim /approot1/k8s/tmp/service/coredns.yaml
clusterIP 参数要和 kubelet 配置文件的 clusterDNS 参数一致
apiVersion: v1kind: ServiceAccountmetadata: name: coredns namespace: kube-system labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: labels: kubernetes.io/bootstrapping: rbac-defaults addonmanager.kubernetes.io/mode: Reconcile name: system:corednsrules:- apiGroups: - "" resources: - endpoints - services - pods - namespaces verbs: - list - watch- apiGroups: - "" resources: - nodes verbs: - get- apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults addonmanager.kubernetes.io/mode: EnsureExists name: system:corednsroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:corednssubjects:- kind: ServiceAccount name: coredns namespace: kube-system---apiVersion: v1kind: ConfigMapmetadata: name: coredns namespace: kube-system labels: addonmanager.kubernetes.io/mode: EnsureExistsdata: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward 、/etc/resolv.conf { max_concurrent 1000 } cache 30 reload loadbalance }---apiVersion: apps/v1kind: Deploymentmetadata: name: coredns namespace: kube-system labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS"spec: replicas: 1 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 selector: matchLabels: k8s-app: kube-dns template: metadata: labels: k8s-app: kube-dns spec: securityContext: seccompProfile: type: RuntimeDefault priorityClassName: system-cluster-critical serviceAccountName: coredns affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchexpressions: - key: k8s-app operator: In values: ["kube-dns"] topologyKey: kubernetes.io/hostname tolerations: - key: "CriticalAddonsOnly" operator: "Exists" nodeSelector: kubernetes.io/os: linux containers: - name: coredns image: docker.io/coredns/coredns:1.8.6 imagePullPolicy: IfNotPresent resources: limits: memory: 300Mi requests: cpu: 100m memory: 70Mi args: [ "-conf", "/etc/coredns/Corefile" ] volumeMounts: - name: config-volume mountPath: /etc/coredns readOnly: true ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP - containerPort: 9153 name: metrics protocol: TCP livenessProbe: httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: httpGet: path: /ready port: 8181 scheme: HTTP securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - all readOnlyRootFilesystem: true dnsPolicy: Default volumes: - name: config-volume configMap: name: coredns items: - key: Corefile path: Corefile---apiVersion: v1kind: Servicemetadata: name: kube-dns namespace: kube-system annotations: prometheus.io/port: "9153" prometheus.io/scrape: "true" labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS"spec: selector: k8s-app: kube-dns clusterIP: 10.88.0.2 ports: - name: dns port: 53 protocol: UDP - name: dns-tcp port: 53 protocol: TCP - name: metrics port: 9153 protocol: TCP
导入 coredns 镜像for i in 192.168.91.19 192.168.91.20;do scp /approot1/k8s/images/coredns-v1.8.6.tar $i:/tmp/ssh $i "ctr -n=k8s.io image import /tmp/coredns-v1.8.6.tar && rm -f /tmp/coredns-v1.8.6.tar"; done
查看镜像
for i in 192.168.91.19 192.168.91.20;do ssh $i "ctr -n=k8s.io image list | grep coredns"; done
在 k8s 中运行 flannel 组件kubectl apply -f /approot1/k8s/tmp/service/coredns.yaml
检查 flannel pod 是否运行成功kubectl get pod -n kube-system | grep coredns
预期输出类似如下结果
因为 coredns yaml 文件内的 replicas 参数是 1 ，因此这里只有一个 pod ，如果改成 2 ，就会出现两个 pod
coredns-5fd74ff788-cddqf 1/1 Running 0 10s
部署 metrics-server 组件 配置 metrics-server yaml 文件vim /approot1/k8s/tmp/service/metrics-server.yaml
apiVersion: v1kind: ServiceAccountmetadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: labels: k8s-app: metrics-server rbac.authorization.k8s.io/aggregate-to-admin: "true" rbac.authorization.k8s.io/aggregate-to-edit: "true" rbac.authorization.k8s.io/aggregate-to-view: "true" name: system:aggregated-metrics-readerrules:- apiGroups: - metrics.k8s.io resources: - pods - nodes verbs: - get - list - watch---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: labels: k8s-app: metrics-server name: system:metrics-serverrules:- apiGroups: - "" resources: - pods - nodes - nodes/stats - namespaces - configmaps verbs: - get - list - watch---apiVersion: rbac.authorization.k8s.io/v1kind: RoleBindingmetadata: labels: k8s-app: metrics-server name: metrics-server-auth-reader namespace: kube-systemroleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: extension-apiserver-authentication-readersubjects:- kind: ServiceAccount name: metrics-server namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: labels: k8s-app: metrics-server name: metrics-server:system:auth-delegatorroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegatorsubjects:- kind: ServiceAccount name: metrics-server namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: labels: k8s-app: metrics-server name: system:metrics-serverroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:metrics-serversubjects:- kind: ServiceAccount name: metrics-server namespace: kube-system---apiVersion: v1kind: Servicemetadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-systemspec: ports: - name: https port: 443 protocol: TCP targetPort: https selector: k8s-app: metrics-server---apiVersion: apps/v1kind: Deploymentmetadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-systemspec: selector: matchLabels: k8s-app: metrics-server strategy: rollingUpdate: maxUnavailable: 0 template: metadata: labels: k8s-app: metrics-server spec: containers: - args: - --cert-dir=/tmp - --secure-port=4443 - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --kubelet-insecure-tls - --kubelet-use-node-status-port - --metric-resolution=15s image: k8s.gcr.io/metrics-server/metrics-server:v0.5.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /livez port: https scheme: HTTPS periodSeconds: 10 name: metrics-server ports: - containerPort: 4443 name: https protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /readyz port: https scheme: HTTPS initialDelaySeconds: 20 periodSeconds: 10 resources: requests: cpu: 100m memory: 200Mi securityContext: readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000 volumeMounts: - mountPath: /tmp name: tmp-dir nodeSelector: kubernetes.io/os: linux priorityClassName: system-cluster-critical serviceAccountName: metrics-server volumes: - emptyDir: {} name: tmp-dir---apiVersion: apiregistration.k8s.io/v1kind: APIServicemetadata: labels: k8s-app: metrics-server name: v1beta1.metrics.k8s.iospec: group: metrics.k8s.io groupPriorityMinimum: 100 insecureSkipTLSVerify: true service: name: metrics-server namespace: kube-system version: v1beta1 versionPriority: 100
导入 metrics-server 镜像for i in 192.168.91.19 192.168.91.20;do scp /approot1/k8s/images/metrics-server-v0.5.2.tar $i:/tmp/ssh $i "ctr -n=k8s.io image import /tmp/metrics-server-v0.5.2.tar && rm -f /tmp/metrics-server-v0.5.2.tar"; done
查看镜像
for i in 192.168.91.19 192.168.91.20;do ssh $i "ctr -n=k8s.io image list | grep metrics-server"; done
在 k8s 中运行 metrics-server 组件kubectl apply -f /approot1/k8s/tmp/service/metrics-server.yaml
检查 metrics-server pod 是否运行成功kubectl get pod -n kube-system | grep metrics-server
预期输出类似如下结果
metrics-server-6c95598969-qnc76 1/1 Running 0 71s
验证 metrics-server 功能
查看节点资源使用情况
kubectl top node
预期输出类似如下结果
metrics-server 启动会偏慢，速度取决于机器配置，如果输出 is not yet 或者 is not ready 就等一会再执行一次 kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%192.168.91.19 285m 4% 2513Mi 32%192.168.91.20 71m 3% 792Mi 21%
查看指定 namespace 的 pod 资源使用情况
kubectl top pod -n kube-system
预期输出类似如下结果
NAME CPU(cores) MEMORY(bytes)coredns-5fd74ff788-cddqf 11m 18Mikube-flannel-ds-86rrv 4m 18Mikube-flannel-ds-bkgzx 6m 22Mikube-flannel-ds-v25xc 6m 22Mimetrics-server-6c95598969-qnc76 6m 22Mi
部署 dashboard 组件 配置 dashboard yaml 文件vim /approot1/k8s/tmp/service/dashboard.yaml
---apiVersion: v1kind: ServiceAccountmetadata: name: admin-user namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: admin-userroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects:- kind: ServiceAccount name: admin-user namespace: kube-system---apiVersion: v1kind: ServiceAccountmetadata: name: dashboard-read-user namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: dashboard-read-bindingroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: dashboard-read-clusterrolesubjects:- kind: ServiceAccount name: dashboard-read-user namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: dashboard-read-clusterrolerules:- apiGroups: - "" resources: - configmaps - endpoints - nodes - persistentvolumes - persistentvolumeclaims - persistentvolumeclaims/status - pods - replicationcontrollers - replicationcontrollers/scale - serviceaccounts - services - services/status verbs: - get - list - watch- apiGroups: - "" resources: - bindings - events - limitranges - namespaces/status - pods/log - pods/status - replicationcontrollers/status - resourcequotas - resourcequotas/status verbs: - get - list - watch- apiGroups: - "" resources: - namespaces verbs: - get - list - watch- apiGroups: - apps resources: - controllerrevisions - daemonsets - daemonsets/status - deployments - deployments/scale - deployments/status - replicasets - replicasets/scale - replicasets/status - statefulsets - statefulsets/scale - statefulsets/status verbs: - get - list - watch- apiGroups: - autoscaling resources: - horizontalpodautoscalers - horizontalpodautoscalers/status verbs: - get - list - watch- apiGroups: - batch resources: - cronjobs - cronjobs/status - jobs - jobs/status verbs: - get - list - watch- apiGroups: - extensions resources: - daemonsets - daemonsets/status - deployments - deployments/scale - deployments/status - ingresses - ingresses/status - replicasets - replicasets/scale - replicasets/status - replicationcontrollers/scale verbs: - get - list - watch- apiGroups: - policy resources: - poddisruptionbudgets - poddisruptionbudgets/status verbs: - get - list - watch- apiGroups: - networking.k8s.io resources: - ingresses - ingresses/status - networkpolicies verbs: - get - list - watch- apiGroups: - storage.k8s.io resources: - storageclasses - volumeattachments verbs: - get - list - watch- apiGroups: - rbac.authorization.k8s.io resources: - clusterrolebindings - clusterroles - roles - rolebindings verbs: - get - list - watch---apiVersion: v1kind: ServiceAccountmetadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-system---kind: ServiceapiVersion: v1metadata: labels: k8s-app: kubernetes-dashboard kubernetes.io/cluster-service: "true" name: kubernetes-dashboard namespace: kube-systemspec: ports: - port: 443 targetPort: 8443 selector: k8s-app: kubernetes-dashboard type: NodePort---apiVersion: v1kind: Secretmetadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-certs namespace: kube-systemtype: Opaque---apiVersion: v1kind: Secretmetadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-csrf namespace: kube-systemtype: Opaquedata: csrf: ""---apiVersion: v1kind: Secretmetadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-key-holder namespace: kube-systemtype: Opaque---kind: ConfigMapapiVersion: v1metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-settings namespace: kube-system---kind: RoleapiVersion: rbac.authorization.k8s.io/v1metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-systemrules: # Allow Dashboard to get, update and delete Dashboard exclusive secrets. - apiGroups: [""] resources: ["secrets"] resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"] verbs: ["get", "update", "delete"] # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map. - apiGroups: [""] resources: ["configmaps"] resourceNames: ["kubernetes-dashboard-settings"] verbs: ["get", "update"] # Allow Dashboard to get metrics. - apiGroups: [""] resources: ["services"] resourceNames: ["heapster", "dashboard-metrics-scraper"] verbs: ["proxy"] - apiGroups: [""] resources: ["services/proxy"] resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"] verbs: ["get"]---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboardrules: # Allow Metrics Scraper to get metrics from the Metrics server - apiGroups: ["metrics.k8s.io"] resources: ["pods", "nodes"] verbs: ["get", "list", "watch"]---apiVersion: rbac.authorization.k8s.io/v1kind: RoleBindingmetadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-systemroleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubernetes-dashboardsubjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: kubernetes-dashboardroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kubernetes-dashboardsubjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kube-system---kind: DeploymentapiVersion: apps/v1metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-systemspec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: kubernetes-dashboard template: metadata: labels: k8s-app: kubernetes-dashboard spec: containers: - name: kubernetes-dashboard image: kubernetesui/dashboard:v2.4.0 imagePullPolicy: IfNotPresent ports: - containerPort: 8443 protocol: TCP args: - --auto-generate-certificates - --namespace=kube-system - --token-ttl=1800 - --sidecar-host=http://dashboard-metrics-scraper:8000 # Uncomment the following line to manually specify Kubernetes API server Host # If not specified, Dashboard will attempt to auto discover the API server and connect # to it、Uncomment only if the default does not work. # - --apiserver-host=http://my-address:port volumeMounts: - name: kubernetes-dashboard-certs mountPath: /certs # Create on-disk volume to store exec logs - mountPath: /tmp name: tmp-volume livenessProbe: httpGet: scheme: HTTPS path: / port: 8443 initialDelaySeconds: 30 timeoutSeconds: 30 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 volumes: - name: kubernetes-dashboard-certs secret: secretName: kubernetes-dashboard-certs - name: tmp-volume emptyDir: {} serviceAccountName: kubernetes-dashboard nodeSelector: "kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule---kind: ServiceapiVersion: v1metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kube-systemspec: ports: - port: 8000 targetPort: 8000 selector: k8s-app: dashboard-metrics-scraper---kind: DeploymentapiVersion: apps/v1metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kube-systemspec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: dashboard-metrics-scraper template: metadata: labels: k8s-app: dashboard-metrics-scraper spec: securityContext: seccompProfile: type: RuntimeDefault containers: - name: dashboard-metrics-scraper image: kubernetesui/metrics-scraper:v1.0.7 imagePullPolicy: IfNotPresent ports: - containerPort: 8000 protocol: TCP livenessProbe: httpGet: scheme: HTTP path: / port: 8000 initialDelaySeconds: 30 timeoutSeconds: 30 volumeMounts: - mountPath: /tmp name: tmp-volume securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 serviceAccountName: kubernetes-dashboard nodeSelector: "kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule volumes: - name: tmp-volume emptyDir: {}
导入 dashboard 镜像for i in 192.168.91.19 192.168.91.20;do scp /approot1/k8s/images/dashboard-*.tar $i:/tmp/ssh $i "ctr -n=k8s.io image import /tmp/dashboard-v2.4.0.tar && rm -f /tmp/dashboard-v2.4.0.tar"; ssh $i "ctr -n=k8s.io image import /tmp/dashboard-metrics-scraper-v1.0.7.tar && rm -f /tmp/dashboard-metrics-scraper-v1.0.7.tar"; done
查看镜像
for i in 192.168.91.19 192.168.91.20;do ssh $i "ctr -n=k8s.io image list | egrep 'dashboard|metrics-scraper'"; done
在 k8s 中运行 dashboard 组件kubectl apply -f /approot1/k8s/tmp/service/dashboard.yaml
检查 dashboard pod 是否运行成功kubectl get pod -n kube-system | grep dashboard
预期输出类似如下结果
dashboard-metrics-scraper-799d786dbf-v28pm 1/1 Running 0 2m55skubernetes-dashboard-9f8c8b989-rhb7z 1/1 Running 0 2m55s
查看 dashboard 访问端口在 service 当中没有指定 dashboard 的访问端口，所以需要自己获取，也可以修改 yaml 文件指定访问端口
预期输出类似如下结果
我这边是将 30210 端口映射给 pod 的 443 端口
kubernetes-dashboard NodePort 10.88.127.68
查看 dashboard 登录 token根据得到的端口访问 dashboard 页面，例如： https://192.168.91.19:30210
获取 token 文件名称
kubectl get secrets -n kube-system | grep admin
预期输出类似如下结果
admin-user-token-zvrst kubernetes.io/service-account-token 3 9m2s
获取 token 内容
kubectl get secrets -n kube-system admin-user-token-zvrst -o jsonpath={.data.token}|base64 -d
预期输出类似如下结果
eyJhbGciOiJSUzI1NiIsImtpZCI6InA4M1lhZVgwNkJtekhUd3Vqdm9vTE1ma1JYQ1ZuZ3c3ZE1WZmJhUXR4bUUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXp2cnN0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJhYTE3NTg1ZC1hM2JiLTQ0YWYtOWNhZS0yNjQ5YzA0YThmZWYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.K2o9p5St9tvIbXk7mCQCwsZQV11zICwN-JXhRv1hAnc9KFcAcDOiO4NxIeicvC2H9tHQBIJsREowVwY3yGWHj_MQa57EdBNWMrN1hJ5u-XzpzJ6JbQxns8ZBrCpIR8Fxt468rpTyMyqsO2UBo-oXQ0_ZXKss6X6jjxtGLCQFkz1ZfFTQW3n49L4ENzW40sSj4dnaX-PsmosVOpsKRHa8TPndusAT-58aujcqt31Z77C4M13X_vAdjyDLK9r5ZXwV2ryOdONwJye_VtXXrExBt9FWYtLGCQjKn41pwXqEfidT8cY6xbA7XgUVTr9miAmZ-jf1UeEw-nm8FOw9Bb5v6A
到此，基于 containerd 二进制部署 k8s v1.23.3 就结束了