Youluo's Blog

Appearance

部署目标 ​

  • 在4台主机上部署分布式k8s集群，模拟真实生产环境使用需求。具体的要求参考官方文档。

    最新版本为v1.30。我们安装前两个版本，即 v1.28.10。

环境说明 ​

IPhostname硬件参数操作系统版本角色
192.168.107.8k8s-master013C/4G/50GCentOS 7.6.1810master
192.168.107.9k8s-node013C/4G/50GCentOS 7.6.1810node
192.168.107.10k8s-node023C/4G/50GCentOS 7.6.1810node
192.168.107.11k8s-node033C/4G/50GCentOS 7.6.1810node

基础环境准备 ​

root账号操作

  • 设置主机名（差异操作）

    shell
    # 192.168.107.8
hostnamectl set-hostname k8s-master01
# 192.168.107.9
hostnamectl set-hostname k8s-node01
# 192.168.107.10
hostnamectl set-hostname k8s-node02
# 192.168.107.11
hostnamectl set-hostname k8s-node03

  • 配置/etc/hosts

    shell
    192.168.107.8    k8s-master01.ilinux.io k8s-master01 k8s-api.ilinux.io ck01
192.168.107.9    k8s-node01.ilinux.io k8s-node01 ck02
192.168.107.10   k8s-node02.ilinux.io k8s-node02 ck03
192.168.107.11   k8s-node03.ilinux.io k8s-node03 ck04


ping -c 2 k8s-master01
ping -c 2 k8s-node01
ping -c 2 k8s-node02
ping -c 2 k8s-node03

  • 主机时间同步

    shell
    systemctl start chronyd
systemctl enable chronyd
chronyc sources -v

  • 关闭防火墙

    此处选择关闭，生产环境不要关闭

    shell
    # 关闭firewalld
systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld

# 2 关闭iptables服务
systemctl stop iptables
systemctl disable iptables
systemctl status iptables

  • 关闭 selinux

    shell
    setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config 
cat /etc/selinux/config

  • 禁用swap设备

    shell
    # 临时禁止
swapoff -a 
# 永久禁止
sed -i '/ swap / s/^/#/' /etc/fstab
#vim /etc/fstab
# 注释所有文件系统类型为swap的配置行

  • 确保MAC地址唯一

    shell
    ip link show ens33

  • 更改网络配置

    shell
    # 网络配置文件
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
EOF

#加载网桥过滤模块,执行该命令 如果不执行就会在应用k8s.conf时出现加载错误
modprobe br_netfilter
#应用配置文件
sysctl -p /etc/sysctl.d/k8s.conf 
# 查看网桥过滤模块是否加载成功
lsmod | grep br_netfilter

安装 ​

安装Docker ​

shell
cat /etc/redhat-release
uname -r
yum remove docker  docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine docker-ce
yum -y install gcc gcc-c++ yum-utils
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum makecache fast
yum install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

mkdir -p /etc/docker
# 修改cgroup驱动为systemd，这是k8s官方推荐，且不使用这个后续可能会出现一些问题
cat > /etc/docker/daemon.conf <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

cat /etc/docker/daemon.conf

systemctl restart docker
systemctl enable docker
systemctl status docker

docker version

此处按照 /etc/docker/daemon.json，出错。解决参考：记一次centos7重启后docker无法启动的问题

安装k8s ​

安装 ​

shell
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF

yum clean all && yum makecache

# 查看所有的可用版本
yum list kubeadm kubelet kubectl --showduplicates | sort -r

# 安装kubeadm、kubelet和kubectl
yum install kubeadm kubelet kubectl -y
# yum install kubeadm kubelet kubectl -y --disableexcludes=kubernetes

# 查看kubeadm版本
kubeadm version
### kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.2", GitCommit:"89a4ea3e1e4ddd7f7572286090359983e0387b2f", GitTreeState:"clean", BuildDate:"2023-09-13T09:34:32Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}

# 设置kubelet开机自启
systemctl enable kubelet && systemctl start kubelet
systemctl status kubelet

# 验证kubectl版本
kubectl version
# 验证kubeadm版本
yum info kubeadm

Master节点初始化 ​

在k8s-master01上操作，使用root账号

配置 ​

shell
# 查看所需要镜像
kubeadm config images list

# 查看阿里云镜像仓库地址中的镜像列表
kubeadm config images list  --image-repository registry.aliyuncs.com/google_containers

# 拉取阿里云镜像仓库地址中的镜像
sed -i -r '/cri/s/(.*)/#\1/' /etc/containerd/config.toml
systemctl restart containerd
kubeadm config images pull  --image-repository registry.aliyuncs.com/google_containers

初始化操作 ​

shell
mkdir -p /etc/systemd/system/kubelet.service.d
cat <<EOF > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
EOF


kubeadm init -v=5\
  --apiserver-advertise-address=192.168.107.8 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.28.10 \
  --service-cidr=10.96.0.0/12 \
  --pod-network-cidr=10.244.0.0/16 \
  --ignore-preflight-errors=all

–apiserver-advertise-address 集群通告地址(master 机器IP，这里用的万兆网) –image-repository 由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址 –kubernetes-version K8s版本，与上面安装的一致 –service-cidr 集群内部虚拟网络，Pod统一访问入口，可以不用更改，直接用上面的参数 –pod-network-cidr Pod网络，与下面部署的CNI网络组件yaml中保持一致，可以不用更改，直接用上面的参数

遇到的错误：

（可选）重置kubeadm ​

如果初始化失败，或者需要重新初始化，可执行以下命令进行重置 重置adm

l
kubeadm reset

移除相关文件

l
rm -fr ~/.kube/  /etc/kubernetes/* var/lib/etcd/*

参考资料 ​

shell
# https://blog.csdn.net/m0_74226049/article/details/135899829
containerd config default > /etc/containerd/config.toml
cat -n /etc/containerd/config.toml | grep -B 12 'SystemdCgroup'
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
cat -n /etc/containerd/config.toml | grep -B 12 'SystemdCgroup'


systemctl restart containerd
systemctl enable containerd
systemctl status containerd

kubeadm reset
rm -fr ~/.kube/  /etc/kubernetes/* var/lib/etcd/*
shell
# https://cloud.tencent.com/developer/ask/sof/107741164
systemctl stop kubelet
systemctl stop docker 

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
swapoff -a

systemctl start --now docker
systemctl start --now kubelet
systemctl status kubelet

卸载 ​

shell
yum remove -y kubeadm kubectl kubelet kubernetes-cni kube* 
yum remove docker  docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
yum autoremove -y
rm -rf /etc/systemd/system/kubelet.service
rm -rf /etc/systemd/system/kube*
rm -rf ~/.kube
rm -rf /etc/kubernetes/
rm -rf /var/lib/kube*

参考 https://blog.csdn.net/m0_51720581/article/details/131153894

shell
systemctl start docker
systemctl enable docker
systemctl status docker


# 配置Docker使用systemd作为默认Cgroup驱动，配置之后需要重启docker
cat <<EOF > /etc/docker/daemon.json
{
	"registry-mirrors": [
        "http://hub-mirror.c.163.com",
        "https://docker.mirrors.ustc.edu.cn",
        "https://registry.docker-cn.com"
    ],
	"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

cat /etc/docker/daemon.json

systemctl restart docker
shell
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum install -y kubelet-1.23.6 kubeadm-1.23.6 kubectl-1.23.6


systemctl enable kubelet


docker pull coredns/coredns:1.8.4
docker tag coredns/coredns:1.8.4 registry.aliyuncs.com/google_containers/coredns:v1.8.4

  • master上执行初始化

    shell
    kubeadm init \
  --apiserver-advertise-address=192.168.107.8 \
  --image-repository registry.aliyuncs.com/google_containers \
  --service-cidr=10.1.0.0/16 \
  --pod-network-cidr=10.244.0.0/16 \
  --kubernetes-version=v1.23.6

    显示如下结果：

    shell
    [addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.107.8:6443 --token nbhkt4.lky1xk2dr2mvqimm \
	--discovery-token-ca-cert-hash sha256:f2e427fc2cfa57b11555926c9cc9ad94ba147fc0c1a04deb1e483f3cdbefea2e

    • 3个 worker node执行

    • master上执行

      shell
      mkdir -p $HOME/.kube

    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u)😒(id -g) $HOME/.kube/config

    
[root@k8s-master01 ~]# kubectl get node
NAME           STATUS     ROLES                  AGE     VERSION
k8s-master01   NotReady   control-plane,master   7m19s   v1.23.6
k8s-node01     NotReady   <none>                 2m27s   v1.23.6
k8s-node02     NotReady   <none>                 2m13s   v1.23.6
k8s-node03     NotReady   <none>                 2m3s    v1.23.6

    • 安装网络插件，master上执行：

      shell
      wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.0/manifests/tigera-operator.yaml

wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.0/manifests/custom-resources.yaml

kubectl create -f tigera-operator.yaml

    kubectl create -f custom-resources.yaml

    --- ​

    wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml kubectl apply -f kube-flannel.yml

    查看部署进度 ​

    watch kubectl get pods -n calico-system watch kubectl get pods -n kube-flannel

    [root@k8s-master01 ~]# kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-flannel kube-flannel-ds-pg7kl 0/1 Init:ImagePullBackOff 0 6m3s kube-system coredns-6d8c4cb4d-b5ltb 0/1 Pending 0 9m7s kube-system coredns-6d8c4cb4d-nh57v 0/1 Pending 0 9m7s kube-system etcd-k8s-master01 1/1 Running 1 9m22s kube-system kube-apiserver-k8s-master01 1/1 Running 0 9m24s kube-system kube-controller-manager-k8s-master01 1/1 Running 0 9m24s kube-system kube-proxy-jk9vg 1/1 Running 0 9m7s kube-system kube-scheduler-k8s-master01 1/1 Running 0 9m23s [root@k8s-master01 ~]# wget https://github.com/flannel-io/flannel/releases/download/v0.25.4/flanneld-v0.25.4-amd64.docker

    docker load -i flanneld-v0.25.4-amd64.docker

    docker tag quay.io/coreos/flannel:v0.25.4-amd64 docker.io/flannel/flannel:v0.25.4 docker tag quay.io/coreos/flannel:v0.25.4-amd64 docker.io/flannel/flannel:v0.25.4

    重新执行 ​

    kubectl delete -f kube-flannel.yml kubectl apply -f kube-flannel.yml

    


-

基于 知识共享 CC BY-NC-SA 许可发布

版权所有 © 2019-2025 叶莜落