k8s1.20二进制包安装

2023-07-26 15:46 由 Linuxbugs 发表于 #其他

集群环境准备

主机规划

主机IP地址	主机名	主机角色	主机配置	软件列表
10.58.32.31	k8s-master01	master,LB	2C4G	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker，haproxy、keepalived
10.58.32.32	k8s-master02	master，LB	2C4G	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker，haproxy、keepalived
10.58.32.33	k8s-master03	master，LB	2C4G	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker，haproxy、keepalived
10.58.32.34	k8s-node01	worker	1C2G	kubelet、kube-proxy、docker
192.168.10.100	/	VIP(虚拟IP)	/

软件版本

软件名称	版本	备注
Rocky8	kernel版本：5.4
kubernetes	v1.20.15
etcd	v3.5.2	最新版本
calico	v3.4.13
coredns	v1.8.4
docker	1.19
haproxy	5.18	YUM源默认
keepalived	3.5	YUM源默认

网络分配

网络名称	网段	备注
Node网络	10.58.32.30~60/24
Service网络	10.96.0.0/16
Pod网络	172.16.0.0/12

参考：25.基于二进制包安装kubernetes v1.21 --集群部署(二) - 掘金 (juejin.cn)

安装Rocky 8

Vmware 安装

Node网络用的是NAT模式，然后又加了一个网卡，用于仅主机模式

系统初始化

升级系统

yum update -y --exclude=kernel*

更新内核

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum install  -y https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
yum install -y https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm

yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
yum --enablerepo=elrepo-kernel install kernel-lt -y

查看目前可用内核

awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg

使用序号为0的内核，序号0是前面查出来的可用内核编号

grub2-set-default 0

生成 grub 配置文件并重启

grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

修改网络IP

保证每台机器的网络UUID不同, 去掉UUID

vim /etc/sysconfig/network-scripts/ifcfg-ens192

... 

BOOTPROTO=static
...
ONBOOT=yes
IPADDR=10.58.32.31
NETMASK=255.255.255.0

nmcli connection reload

基础配置

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

# 安装vim
yum install -y vim lrzsz

# 关闭selinux
sed -i 's/SELINUX=.*$/SELINUX=disabled/g'  /etc/selinux/config

# 关闭swap
swapoff -a 
sed -ri 's/.*swap.*/#&/' /etc/fstab

# 设置 open file
cat >> /etc/security/limits.conf << EOF

* hard noproc 65536
* soft noproc 65536
* hard nofile 65536
* soft nofile 65536
EOF

编辑hosts文件

cat >>  /etc/hosts <<EOF

10.58.32.30 k8s-master-lb
10.58.32.31 k8s-master01
10.58.32.32 k8s-master02
10.58.32.33 k8s-master03
10.58.32.34 k8s-node01
10.58.32.35 k8s-node02
EOF

ntp配置

yum install epel-release -y
yum -ivh https://mirrors.wlnmp.com/rockylinux/wlnmp-release-rocky-8.noarch.rpm
yum install wntp -y

ntpdate time2.aliyun.com

crontab -e
*/5 * * * * ntpdate time2.aliyun.com

cat >> /etc/rc.local << EOF

ntpdate time2.aliyun.com
EOF

chmod +x /etc/rc.local

系统优化

cat > /etc/sysctl.d/k8s_better.conf << EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF

sysctl -p /etc/sysctl.d/k8s_better.conf

modprobe br_netfilter
lsmod |grep conntrack
modprobe ip_conntrack

安装IPVS

yum install -y conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git

## 开启ipvs 转发
modprobe br_netfilter 

cat > /etc/sysconfig/modules/ipvs.modules << EOF 
#!/bin/bash 
modprobe -- ip_vs 
modprobe -- ip_vs_rr 
modprobe -- ip_vs_lc 
modprobe -- ip_vs_lblc 
modprobe -- ip_vs_dh
modprobe -- ip_vs_fo 
modprobe -- ip_vs_nq 
modprobe -- ip_vs_wrr 
modprobe -- ip_vs_sh 
modprobe -- ip_vs_sed 
modprobe -- ip_vs_ftp
modprobe -- nf_conntrack
modprobe -- ip_tables
modprobe -- ip_set
modprobe -- xt_set
modprobe -- ipt_set
modprobe -- ipt_rpfilter
modprobe -- ipt_REJECT
modprobe -- ipip
EOF 

chmod 755 /etc/sysconfig/modules/ipvs.modules 
bash /etc/sysconfig/modules/ipvs.modules 
lsmod | grep -e ip_vs -e nf_conntrack

特别注意，如果内核是4.19以下的，nf_conntrack 要改为nf_conntrack_ipv4

安装docker

yum install -y yum-utils device-mapper-persistent-data lvm2
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo 

yum list docker-ce --showduplicates
yum install docker-ce-19.03.13-3.el8

修改从cgroup方式

cat > /etc/docker/daemon.json <<EOF
{
    "registry-mirrors":[
        "https://registry.docker-cn.com",
        "https://hub-mirrors.c.163.com",
        "https://docker.mirrors.utsc.edu.cn"
    ],
    "exec-opts":["native.cgroupdriver=systemd"],
    "max-concurrent-downloads": 10,
    "max-concurrent-uploads": 5,
    "log-opts":{
        "max-size": "300m",
        "max-file":"2"
    },
    "live-restore": true
}
EOF

然后

systemctl daemon-reload && systemctl restart docker && systemctl enable docker

master01打通ssh到其他节点

master01节点执行

ssh-keygen

# 执行所有节点
for node in 10.58.32.{31..35};do
    ssh-copy-id root@${node}
done

下载k8s-server

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md #server-binaries

下载kubernetes-server-linux-amd64.tar.gz 并上传到所有节点

tar -xvf kubernetes-server-linux-amd64.tar.gz
mv kubernetes /

vim /etc/profile.d/k8s.sh

export K8S_HOME=/kubernetes
export K8S_SERVER=$K8S_HOME/server
export PATH=$PATH:$K8S_SERVER/bin

chmod +x /etc/profile.d/k8s.sh
ssource /etc/profile.d/k8s.sh

生成证书

master01节点下载cfssl

创建工作目录
mkdir /data/k8s-work
cd /data/k8s-work

下载地址： https://github.com/cloudflare/cfssl/releases?page=1
cfssl_1.6.3_linux_amd64
cfssljson_1.6.3_linux_amd64

移动到指定目录

chmod +x cfssl_1.6.3_linux_amd64 cfssljson_1.6.3_linux_amd64
cp cfssl_1.6.3_linux_amd64 /usr/local/bin/cfssl
cp cfssljson_1.6.3_linux_amd64 /usr/local/bin/cfssljson

创建证书保存目录

# 所有节点
mkdir -p /etc/etcd/ssl
mkdir -p /etc/kubernetes/pki

下载开源工具包

git clone https://github.com/dotbalo/k8s-ha-install.git

cd k8s-ha-install
git branch -a
git checkout remotes/origin/manual-installation-v1.20.x

生成etcd证书

cd /root/k8s-ha-install/pki

[root@k8s-master01 pki]# ls
admin-csr.json ca-csr.json  front-proxy-ca-csr.json  kube-proxy-csr.json apiserver-csr.json  etcd-ca-csr.json  front-proxy-client-csr.json  manager-csr.json  ca-config.json etcd-csr.json  kubelet-csr.json  scheduler-csr.json

etcd的证书和k8s的证书是完全独立的，没有任何关系

CSR文件：证书签名请求文件

# ca证书
cfssl gencert -initca etcd-ca-csr.json|cfssljson -bare /etc/etcd/ssl/etcd-ca
# 客户端证书
cfssl gencert \
-ca=/etc/etcd/ssl/etcd-ca.pem \
-ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
-config=ca-config.json \
-hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,10.58.32.31,10.58.32.32,10.58.32.33 \
-profile=kubernetes \
etcd-csr.json|cfssljson -bare /etc/etcd/ssl/etcd

生成kubernetes证书

# ca证书
cfssl gencert -initca ca-csr.json|cfssljson -bare /etc/kubernetes/pki/ca

# apiserver需要证书
# 10.96.0是k8s service的网段，如果需要更改k8s service的网段，就需要更改10.96.0.1
# 10.58.32.30 vip网段
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-hostname=10.96.0.1,10.58.32.30,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,10.58.32.31,10.58.32.32,10.58.32.33 \
-profile=kubernetes \
apiserver-csr.json|cfssljson -bare /etc/kubernetes/pki/apiserver


# 生成controller-manage的证书
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
manager-csr.json|cfssljson -bare /etc/kubernetes/pki/controller-manager

# 生成scheduler证书
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
scheduler-csr.json|cfssljson -bare /etc/kubernetes/pki/scheduler


# 生成admin的证书
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
admin-csr.json|cfssljson -bare /etc/kubernetes/pki/admin

生成apiserver聚合证书

# ca证书
cfssl gencert -initca front-proxy-ca-csr.json|cfssljson -bare /etc/kubernetes/pki/front-proxy-ca

# 生成聚合证书
# Requestheader-client-xxx    requestheader-allowed-xxx:aggerator
cfssl gencert \
-ca=/etc/kubernetes/pki/front-proxy-ca.pem \
-ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
front-proxy-client-csr.json|cfssljson -bare /etc/kubernetes/pki/front-proxy-client

生成kubeconfig文件

生成controller的kubeconfig

# 设置集群项
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.58.32.30:8443 \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 设置用户项
kubectl config set-credentials system:kube-contoller-manager \
--client-certificate=/etc/kubernetes/pki/controller-manager.pem \
--client-key=/etc/kubernetes/pki/controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 设置环境项，上下文
kubectl config set-context system:kube-contoller-manager@kubernetes \
--cluster=kubernetes \
--user=system:kube-contoller-manager \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 将上面的环境设置为默认环境
kubectl config use-context system:kube-contoller-manager@kubernetes \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

生成schedule的kubeconfig

kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.58.32.30:8443 \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# 设置用户项
kubectl config set-credentials system:kube-scheduler \
--client-certificate=/etc/kubernetes/pki/scheduler.pem \
--client-key=/etc/kubernetes/pki/scheduler-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# 设置环境项，上下文
kubectl config set-context system:kube-scheduler@kubernetes \
--cluster=kubernetes \
--user=system:kube-scheduler \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# 将上面的环境设置为默认环境
kubectl config use-context system:kube-scheduler@kubernetes \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

生成admin的kubeconfig

kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.58.32.30:8443 \
--kubeconfig=/etc/kubernetes/admin.kubeconfig

# 设置用户项
kubectl config set-credentials kubernetes-admin \
--client-certificate=/etc/kubernetes/pki/admin.pem \
--client-key=/etc/kubernetes/pki/admin-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/admin.kubeconfig

# 设置环境项，上下文
kubectl config set-context kubernetes-admin@kubernetes \
--cluster=kubernetes \
--user=kubernetes-admin \
--kubeconfig=/etc/kubernetes/admin.kubeconfig

# 将上面的环境设置为默认环境
kubectl config use-context kubernetes-admin@kubernetes \
--kubeconfig=/etc/kubernetes/admin.kubeconfig

生成ServiceAccount key

openssl genrsa -out /etc/kubernetes/pki/sa.key 2048

openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

拷贝证书到其他master节点

cd /etc/kubernetes/pki
scp -r ./* root@10.58.32.33:/etc/kubernetes/

配置etcd

一定要配置奇数个，否则会产生脑裂

一下操作master节点都要做

下载etcd

wget https://github.com/etcd-io/etcd/releases/download/v3.4.24/etcd-v3.4.24-linux-amd64.tar.gz

tar -xvf etcd-v3.4.24-linux-amd64.tar.gz
mv etcd-v3.4.24-linux-amd64 etcd-v3.4.24
mv etcd-v3.4.24 /kubernetes/

添加到PATH

vim /etc/profile.d/k8s.sh

export K8S_HOME=/kubernetes
export K8S_SERVER=$K8S_HOME/server
export PATH=$PATH:$K8S_SERVER/bin:$K8S_HOME/etcd-v3.4.24

source /etc/profile.d/k8s.sh

配置文件

配置文件每个节点多少有点不同，/etc/etcd/etcd.config.yaml

k8s-master01

name: 'k8s-master01'
data-dir: /kubernetes/etcd-v3.4.24/data
wal-dir: /kubernetes/etcd-v3.4.24/data/wal
snapshot-count: 10000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: https://10.58.32.31:2380
listen-client-urls: https://10.58.32.31:2379,http://127.0.0.1:2379
max-snapshots: 5
max-wals: 5
cors:
initial-advertise-peer-urls: https://10.58.32.31:2380
advertise-client-urls: https://10.58.32.31:2379
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.58.32.31:2380,k8s-master02=https://10.58.32.32:2380,k8s-master03=https://10.58.32.33:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
debug: false
logger: zap
log-outputs: [default]
force-new-cluster: false
auto-compaction-mode: periodic
auto-compaction-retention: "1"

k8s-master02

name: 'k8s-master02'
data-dir: /kubernetes/etcd-v3.4.24/data
wal-dir: /kubernetes/etcd-v3.4.24/data/wal
snapshot-count: 10000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: https://10.58.32.32:2380
listen-client-urls: https://10.58.32.32:2379,http://127.0.0.1:2379
max-snapshots: 5
max-wals: 5
cors:
initial-advertise-peer-urls: https://10.58.32.32:2380
advertise-client-urls: https://10.58.32.32:2379
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.58.32.31:2380,k8s-master02=https://10.58.32.32:2380,k8s-master03=https://10.58.32.33:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
debug: false
logger: zap
log-outputs: [default]
force-new-cluster: false
auto-compaction-mode: periodic
auto-compaction-retention: "1"

k8s-master03

name: 'k8s-master03'
data-dir: /kubernetes/etcd-v3.4.24/data
wal-dir: /kubernetes/etcd-v3.4.24/data/wal
snapshot-count: 10000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: https://10.58.32.33:2380
listen-client-urls: https://10.58.32.33:2379,http://127.0.0.1:2379
max-snapshots: 5
max-wals: 5
cors:
initial-advertise-peer-urls: https://10.58.32.33:2380
advertise-client-urls: https://10.58.32.33:2379
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.58.32.31:2380,k8s-master02=https://10.58.32.32:2380,k8s-master03=https://10.58.32.33:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
debug: false
logger: zap
log-outputs: [default]
force-new-cluster: false
auto-compaction-mode: periodic
auto-compaction-retention: "1"

启动system文件

cat > /etc/systemd/system/etcd.service <<"EOF"
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/kubernetes/etcd-v3.4.24/etcd --config-file=/etc/etcd/etcd.config.yaml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service
EOF

启动etcd集群

systemctl daemon-reload
systemctl enable --now etcd.service
systemctl status etcd

验证集群状态

ETCDCTL_API=3 /kubernetes/etcd-v3.4.24/etcdctl --write-out=table --cacert=/etc/etcd/ssl/etcd-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://10.58.32.31:2379,https://10.58.32.32:2379,https://10.58.32.33:2379 endpoint health

负载均衡器准备

安装haproxy与keepalived

yum -y install haproxy keepalived

HAProxy配置

cat >/etc/haproxy/haproxy.cfg<<"EOF"
global
 maxconn 2000
 ulimit-n 16384
 log 127.0.0.1 local0 err
 stats timeout 30s

defaults
 log global
 mode http
 option httplog
 timeout connect 5000
 timeout client 50000
 timeout server 50000
 timeout http-request 15s
 timeout http-keep-alive 15s

frontend monitor-in
 bind *:33305
 mode http
 option httplog
 monitor-uri /monitor

frontend k8s-master
 bind 0.0.0.0:6443
 bind 127.0.0.1:6443
 mode tcp
 option tcplog
 tcp-request inspect-delay 5s
 default_backend k8s-master

backend k8s-master
 mode tcp
 option tcplog
 option tcp-check
 balance roundrobin
 default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
 server  k8s-master01  10.58.32.31:6443 check
 server  k8s-master02  10.58.32.32:6443 check
 server  k8s-master03  10.58.32.33:6443 check
EOF

Keepalived

主从配置不一致，需要注意。

云上不支持keepalived

k8s-master01

cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
   interval 5
   weight -5
   fall 2 
rise 1
}
vrrp_instance VI_1 {
   state MASTER
   interface ens192
   mcast_src_ip 10.58.32.31
   virtual_router_id 51
   priority 100
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   }
   virtual_ipaddress {
       10.58.32.30
   }
   track_script {
      chk_apiserver
   }
}
EOF

k8s-master02，k8s-master02

cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
  interval 5
   weight -5
   fall 2 
rise 1
}
vrrp_instance VI_1 {
   state BACKUP
   interface ens192
   mcast_src_ip 10.58.32.32,33 # 注意32,33是两个IP后缀
   virtual_router_id 51
   priority 99
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   }
   virtual_ipaddress {
       10.58.32.30
   }
   track_script {
      chk_apiserver
   }
}
EOF

健康检查脚本

master节点均要配置

cat > /etc/keepalived/check_apiserver.sh <<"EOF"
#!/bin/bash
err=0
for k in $(seq 1 3)
do
   check_code=$(pgrep haproxy)
   if [[ $check_code == "" ]]; then
       err=$(expr $err + 1)
       sleep 1
       continue
   else
       err=0
       break
   fi
done

if [[ $err != "0" ]]; then
   echo "systemctl stop keepalived"
   /usr/bin/systemctl stop keepalived
   exit 1
else
   exit 0
fi
EOF

chmod +x /etc/keepalived/check_apiserver.sh

启动服务并验证

systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived

ip address show

k8s组件安装

创建相关目录

mkdir -p /etc/kubernetes/manifests  /var/lib/kubelet /var/log/kubernetes

apiserver部署

该部署所有master节点都要执行

配置启动脚本

注意master节点的advertise-address不同，需要修改

vim /etc/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.service

[Service]
ExecStart=/kubernetes/server/bin/kube-apiserver \
  --v=2 \
  --logtostderr=true \
  --anonymous-auth=false \
  --bind-address=0.0.0.0 \
  --secure-port=6443 \
  --insecure-port=0 \
  --advertise-address=10.58.32.31 \
  --allow-privileged=true \
  --service-cluster-ip-range=10.96.0.0/16 \
  --service-node-port-range=30000-32767 \
  --etcd-servers=https://10.58.32.31:2379,https://10.58.32.32:2379,https://10.58.32.33:2379 \
  --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --client-ca-file=/etc/kubernetes/pki/ca.pem \
  --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
  --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/pki/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/pki/ca-key.pem  \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
  --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --authorization-mode=Node,RBAC \
  --enable-bootstrap-token-auth \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
  --requestheader-allowed-names=aggregator \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
  --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem


Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

启动并验证

systemctl daemon-reload
systemctl enable --now kube-apiserver

systemctl status kube-apiserver

controller-manager部署

配置启动脚本

注意master节点的advertise-address不同，需要修改

vim /etc/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.service

[Service]
ExecStart=/kubernetes/server/bin/kube-controller-manager \
  --v=2 \
  --logtostderr=true \
  --bind-address=127.0.0.1 \
  --root-ca-file=/etc/kubernetes/pki/ca.pem \
  --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
  --service-account-private-key-file=/etc/kubernetes/pki/ca-key.pem \
  --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
  --leader-elect=true \
  --use-service-account-credentials=true \
  --node-monitor-grace-period=40s \
  --node-monitor-period=5s \
  --pod-eviction-timeout=2m0s \
  --controllers=*,bootstrapsigner,tokencleaner \
  --allocate-node-cidrs=true \
  --cluster-cidr=172.16.0.0/12 \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
  --node-cidr-mask-size=24 \
  --tls-cert-file=/etc/kubernetes/pki/controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/pki/controller-manager-key.pem


Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

systemctl daemon-reload 
systemctl enable --now kube-controller-manager
systemctl status kube-controller-manager

schedule部署

cat >/etc/systemd/system/kube-scheduler.service<<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.service


[Service]
ExecStart=/kubernetes/server/bin/kube-scheduler \
	--v=2 \
	--address=127.0.0.1 \
	--leader-elect=true \
	--logtostderr=true \
	--kubeconfig=/etc/kubernetes/scheduler.kubeconfig


Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload 
systemctl enable --now kube-scheduler
systemctl status kube-scheduler

TLS bootstrap自动颁发证书

注意：如果不是高可用集群，vip要改成master01的IP，8443端口，改成默认的6443端口

要特别注意--token参数，是token_id.token_secret

自动给kubectl颁发的证书有效期是5年，到期会自动申请

cd /root/k8s-ha-install/bootstrap

kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.58.32.30:8443 \
--kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig

kubectl config set-credentials kubelet-bootstrap \
--token=c8ad9c.2e4d610cf3e7426e \
--kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig

kubectl config set-context tls-bootstrap-token-user@kubernetes \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig

kubectl config use-context tls-bootstrap-token-user@kubernetes \
--kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig

如果要修改bootstrap.secret.yaml的token-id和token-secret，要保证下面中的字符串保持一致，并且位数一样，还要保证上面命令中token信息和下面一致：

apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-c8ad9c
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  description: "The default bootstrap token generated by 'kubelet '."
  token-id: c8ad9c  # 注意这个
  token-secret: 2e4d610cf3e7426e  # 注意这个
  usage-bootstrap-authentication: "true"
... ...

然后配置kubectl的admin权限

mkdir -p /root/.kube
cp /etc/kubernetes/admin.kubeconfig /root/.kube/config

cd /root/k8s-ha-install/bootstrap
kubectl apply -f bootstrap.secret.yaml

secret/bootstrap-token-c8ad9c created
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-certificate-rotation created
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created

Node节点配置

拷贝证书

# node节点上
$ mkdir -p /etc/kubernetes/pki
$ mkdir -p /etc/etcd/ssl

# master01节点上
cd /etc/etcd/ssl
scp ./* root@10.58.32.34:/etc/etcd/ssl/
scp ./* root@10.58.32.35:/etc/etcd/ssl/
# master01节点上
for i in 10.58.32.{32..35};then do
  scp pki/ca.pem root@$i:/etc/kubernetes/pki/
  scp pki/ca-key.pem root@$i:/etc/kubernetes/pki
  scp pki/front-proxy-ca.pem root@$i:/etc/kubernetes/pki/
  scp kubelet-bootstrap.kubeconfig root@$i:/etc/kubernetes/
done

所有节点执行

mkdir -p /etc/kubernetes/manifests /var/lib/kubelet  /etc/systemd/system/kubelet.service.d /var/log/kubernetes

部署kubelet

vim /etc/kubernetes/kubelet-conf.yaml

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
  - 10.96.0.10
clusterDomain: cluster.local
containerLogMaxSize: 20Mi
containerLogMaxFiles: 5
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableDebuggingHandlers: true
enableControllerAttachDetach: true
enforceNodeAllocatable: 
- pods
eventRecordQPS: 5
eventBurst: 20
evictionHard:
  memory.available:  100Mi
  imagefs.available: 15%
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
fileCheckFrequency: 20s
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIQPS: 5
kubeAPIBurst: 10
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryPullQPS: 5
registryBurst: 10
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 10m
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionldleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

添加kubelet启动脚本

vim /etc/systemd/system/kubelet.service

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/kubernetes/server/bin/kubelet

Restart=always
StartLimitInterval=0
RestartSec=5


[Install]
WantedBy=multi-user.target

vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig "
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin "
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yaml --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.4.1 "
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/kubernetes/server/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

然后

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet

出现如下报错，并且有如下报错内容

Unable to register node "k8s-node02" with API server: Unauthorized

说明是认证的问题，重置TSL bootstrap有效

部署kube-proxy

只在master01节点执行就OK

注意：如果不是高可用集群，10.58.32.30:8443 的IP改成master01的IP，端口改为6443端口

kubectl -n kube-system create serviceaccount kube-proxy

kubectl create clusterrolebinding system:kube-proxy   \
--clusterrole system:node-proxier  \
--serviceaccount kube-system:kube-proxy

SECRET=$(kubectl -n kube-system get sa/kube-proxy \
    --output=jsonpath='{.secrets[0].name}')
JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET \
--output=jsonpath='{.data.token}' | base64 -d)
PKI_DIR=/etc/kubernetes/pki
K8S_DIR=/etc/kubernetes


kubectl config set-cluster kubernetes   \
--certificate-authority=/etc/kubernetes/pki/ca.pem  \
--embed-certs=true   \
--server=https://10.58.32.30:8443   \
--kubeconfig=${K8S_DIR}/kube-proxy.kubeconfig

kubectl config set-credentials kubernetes  \
--token=${JWT_TOKEN}   \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig


kubectl config set-context kubernetes  \
--cluster=kubernetes  \
--user=kubernetes \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig


kubectl config use-context kubernetes  \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

然后切换目录

cd /root/k8s-ha-install/kube-proxy
ll 
-rw-r--r-- 1 root root  813 Mar 14 11:15 kube-proxy.conf
-rw-r--r-- 1 root root  288 Mar 14 11:15 kube-proxy.service
-rw-r--r-- 1 root root 3677 Mar 14 11:15 kube-proxy.yml

如果更改了集群Pod的网段，需要更改kube-proxy/kube-proxy.conf的clusterCIDR: 192.168.0.0/12参数为pod的网段（我们之前设置的）。

vim kube-proxy.conf

apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 172.16.0.0/12
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms

然后是kube-proxy.service，修改可执行文件地址为我们的地址

vim kube-proxy.service

[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/kubernetes/server/bin/kube-proxy \
  --config=/etc/kubernetes/kube-proxy.conf \
  --v=2

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

修改完毕，把他们发送到所有节点

for node in 10.58.32.{31..35};do
scp kube-proxy.conf root@$node:/etc/kubernetes/
scp kube-proxy.service root@$node:/etc/systemd/system/
scp /etc/kubernetes/kube-proxy.kubeconfig root@$node:/etc/kubernetes/
done

然后所有节点启动kube-proxy

systemctl daemon-reload && systemctl enable --now kube-proxy
systemctl status kube-proxy

部署calico

以下步骤只在master执行一次

cd /root/k8s-ha-install/calico

cp calico-etcd.yaml calico-etcd.yaml_backup

ETCD_CA=`cat /etc/etcd/ssl/etcd-ca.pem | base64 | tr -d '\n'`
ETCD_CERT=`cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n'`
ETCD_KEY=`cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n'`

sed -i "s#.*etcd-ca:.*#  etcd-ca: ${ETCD_CA}#g" calico-etcd.yaml
sed -i "s#.*etcd-cert:.*#  etcd-cert: ${ETCD_CERT}#g" calico-etcd.yaml
sed -i "s#.*etcd-key:.*#  etcd-key: ${ETCD_KEY}#g" calico-etcd.yaml

sed -i 's#.*etcd_ca:.*#  etcd_ca: "/calico-secrets/etcd-ca"#g' calico-etcd.yaml
sed -i 's#.*etcd_cert:.*#  etcd_cert: "/calico-secrets/etcd-cert"#g' calico-etcd.yaml
sed -i 's#.*etcd_key:.*#  etcd_key: "/calico-secrets/etcd-key"#g' calico-etcd.yaml

sed -i "s#__ETCD_CA_CERT_FILE__#/etc/etcd/ssl/etcd-ca.pem#g" calico-etcd.yaml
sed -i "s#__ETCD_CERT_FILE__#/etc/etcd/ssl/etcd.pem#g" calico-etcd.yaml
sed -i "s#__ETCD_KEY_FILE__#/etc/etcd/ssl/etcd-key.pem#g" calico-etcd.yaml
sed -i "s#__KUBECONFIG_FILEPATH__#/etc/cni/net.d/calico-kubeconfig#g" calico-etcd.yaml


$ vim calico-etcd.yaml

注意修改如下部分：

kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  etcd_endpoints: "https://10.58.32.31:2379,https://10.58.32.32:2379,https://10.58.32.33:2379"
  etcd_ca: ""
  etcd_cert: ""
  etcd_key: ""
   ... ...
  
  - name: CALICO_IPV4POOL_CIDR
    value: "172.16.0.0/12"  # pod的IP

出现如下报错，可解决

参考该博客解决问题：简单部署3节点Kubernetes(k8s)集群之calico网络组件 - 简书 (jianshu.com)

部署CoreDNS

cd /root/k8s-ha-install/CoreDNS

把网关修改成我们自己的service的网关

vim coredns.yaml

spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.96.0.10
  ports:
  - name: dns

然后安装CoreDNS

kubectl apply -f coredns.yaml

部署Metircs Server

cd  /root/k8s-ha-install/metrics-server-0.4.x
kubectl apply -f comp.yaml

然后查看状态

[root@k8s-master01 metrics-server-0.4.x]# kubectl get po -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
... ...
kube-system   metrics-server-595f65d8d5-txkkl            1/1     Running   0          103s

安装图形化工具

使用kuboard且使用docker话安装

Kuboard介绍 | Kuboard

demo命令(具体查看官方文档)：

docker run -d \
  --restart=unless-stopped \
  --name=kuboard \
  -p 80:80/tcp \
  -p 10081:10081/tcp \
  -e KUBOARD_ENDPOINT="http://10.58.32.32:80" \
  -e KUBOARD_AGENT_SERVER_TCP_PORT="10081" \
  -v /kubernetes/kuboard-data:/data \
  eipwork/kuboard:v3

集群可用性测试

Pod 必须能解析 Servicee
Pod 必须能解析跨 namespace 的 Service
每个节点都必须要能访问 Kubernetes 的kubermetes svc 443 和 kube-dns的 service 53
Pod 和Pod之前要能通

同namespace 能通信
跨 namespace 能通信
跨机器能通信

主机IP一定要用NAT模式，这样才能解析/etc/resolv.conf中的网关

问题可参考：

搭建Kubernetes集群时DNS无法解析问题的处理过程 - 掘金 (juejin.cn)

自定义 DNS 服务 | Kubernetes

调试 DNS 问题 | Kubernetes

kubectl exec busybox  -- nslookup kubernetes.default   # pod必须能解析service
kubectl exec busybox  -- nslookup kube-dns.kube-system # pod必须能解析跨namespace的service

telnet 10.96.0.1 443  # 对应说明3
telnet 10.96.0.10 53

# 对应第三点，进入一个pod，去ping其他机器或者pod
kubectl exec -it busybox -- sh   # 跨namespace/node
/ # ping 10.58.32.31

# 验证deployment部署
kubectl create deploy nginx-deployment --image=nginx --replicas=3
kubectl delete nginx-deployment

生产环境关键性配置

docker容器daemon.json

vim /etc/docker/daemon.json

{
    "registry-mirrors":[
        "https://registry.docker-cn.com",
        "https://hub-mirrors.c.163.com",
        "https://docker.mirrors.utsc.edu.cn"
    ],
    "exec-opts":["native.cgroupdriver=systemd"],
    "max-concurrent-downloads": 10, //下载最大线程数
    "max-concurrent-uploads": 5, // 上传最大线程数
    "log-opts":{
        "max-size": "300m",
        "max-file":"2"
    },
    "live-restore": true    // 重启docker，保证运行中的容器不重启
}

controller-manager证书有效期

增长证书签发有效期

vim /etc/systemd/system/kube-controller-manager.service
# 添加
--experimental-cluster-signing-duration=87600h0m0s \

kubelet加密及资源配置

vim /etc/systemd/system/kube-kubelet.service
# 添加参数
# 增强kubelet加密强度
--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
# 延长镜像下载时间（有些镜像下载很慢）
--image-pull-progress-deadline=30m

对应的配置文件：

allowdUnsafeSysctls 按需配置，会修改内核参数，

kubeReserved，systemReserved是给k8s和系统预留的资源

vim /etc/kubernetes/kubelet-config.yaml

# 添加如下配置
allowdUnsafeSysctls:
- "net.core*"
- "net.ipv4.*"
kubeReserved:
 cpu: "10m"
 memory: 1Gi
 ephemeral-storage: 10Gi
systemReserved:
 cpu: "1"
 memory: 1Gi
 ephemeral-storage: 10Gi

额外的说明

给master和worker节点设置roler

[root@k8s-master01 ~]# kubectl label nodes k8s-master02 node-role.kubernetes.io/master=''
node/k8s-master02 labeled
[root@k8s-master01 ~]# kubectl label nodes k8s-master03 node-role.kubernetes.io/master=''
node/k8s-master03 labeled
[root@k8s-master01 ~]# kubectl label nodes k8s-node01 node-role.kubernetes.io/woker=''
node/k8s-node01 labeled
[root@k8s-master01 ~]# kubectl label nodes k8s-node02 node-role.kubernetes.io/woker=''
node/k8s-node02 labeled


$ kubectl get node --show-labels

openssl 查看证书有效期

$ openssl x509 -in ca.pem -noout dates

TLS bootstrapping颁发续签/kubelet启动过程

官方文档：TLS 启动引导 | Kubernetes

Kubelet 启动
Kubelet 查看 kubelet.kubeconfig,文件，假设没有这个文件
Kubelet会查看本地的 bootstrap.kubeconfig
Kubelet读取 bootstrap.kubeconfig 文件，检索 apiserver 的 ur和一个 token
Kubelet链接 apiserver，使用这个 token 进行认证
- Apiserver会识别 tokenid， apiserver 会查看该 tokenid 对无的bootstrap 的一个 secret
- 找个这个secret 中的一个字段，apiserver ,把这个 token 识别成一个 username，名称是system:bootstrap:<token-id>，属于 system:bootstrappers 这个组，这个组具有申请 csr的权限，该组的权限绑定在一个叫system:nodebootstrapper的 custerroles
  - clusterrole k8s 集群级别的权限控制，它作用整个 k8s 集群
  - clusterrolebinding 集群权限的绑定，它可以帮某个clusterrole绑定到一个用户组或者 seviceaccount
- CSR:相当于一个申请表，可以拿着这个申请表去申请我们的证书。
经过上面的认证，kubelet 就有了一个创建和检索 CSR 的权限
Kubelet 为自己创建一个 CSR，名称为 kubernetesio/kube-apiserver-client-kubelet
CSR 被允许有两种方式:
- K8s管理员使用 kubectl手动的颁发证书
- 如果配置了相关权限，kube-controller-manager 会自动同意。
  - Controller-manager 有一个CSRApprovingContraller。他会校验 kubelet发来的 csr的 username和 group 是否有创建 csr的权限，而且还要验证签发着是否是kubernetes.io/kube-apiserver-client-kubelet
  - Controller-manager 同意 CSR 请求
CSR 被同意后，controller-manager 创建 kubelet 的证书文件
Controller-manager 将证书更新至 CSr的 status 字段
Kubelet 从apiserver 获取证书
Kubelet 从获取到的 key 和证书文件创建 kubelet.kubeconfig
Kubelet 启动完成并正常工作
可选:如果配置了自动续期，kubelet 会在证书文件过期的时候利用之前的 kubeconfig 文件去申请一个新的证书，相当于续约。
新的证书被同意或签发，取决于我们的配置
- Kubelet 创建的 CSR 是属于一个O（orgian）: system:nodes
- CN: system:nodes:主机名

添加默认路由

chmod +x /etc/rc.local

vim /etc/rc.local
route add default gw 10.58.32.2 ens192

resolv.conf不断被修改

解决方法
/etc/sysconfig/network-scripts/ifcfg-eth0或者自己网卡配置文件增加一行：PEERDNS=no，然后重新启动网络即可。修改后，重启主机等操作便不会使/etc/resolv.conf被dhclient修改

# Generated by NetworkManager
search localdomain
nameserver 10.58.32.2

关于service注册服务

Linux配置service服务 - 裸睡的猪 - 博客园 (cnblogs.com)

Linux下服务的管理_vendor preset_liuchonghua的博客-CSDN博客

Systemd 入门教程：命令篇 - 阮一峰的网络日志 (ruanyifeng.com)

多网卡集群修改calico通信网卡

vim calico.yaml

env:
  ... ...
  - name: IP_AUTODETECTION_METHOD
    value: "interface=ens33"
  ... ...

热门相关：总裁别再玩了闺范上神来了戏精老公今天作死没戏精老公今天作死没