Kubernetes部署Etcd集群

文章目录
[隐藏]

1. 部署Etcd集群

Etcd 是一个分布式键值存储系统,Kubernetes使用Etcd进行数据存储,所以先准备一个Etcd数据库,为解决Etcd单点故障,应采用集群方式部署,这里使用3台组建集群,可容忍1台机器故障,当然,你也可以使用5台组建集群,可容忍2台机器故障。

注:为了节省机器,这里与K8s节点机器复用,也可以独立于k8s集群之外部署,只要apiserver能连接到就行。

2. 准备cfssl证书生成工具

cfssl是一个开源的证书管理工具,使用json文件生成证书,相比openssl更方便使用。

在k8s-master1节点执行—下载文件

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

在k8s-master1节点执行—赋值执行权限

chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64

在k8s-master1节点执行—将文件剪切到指定目录下并改名

mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

3. 生成Etcd证书—自签证书颁发机构(CA)

在k8s-master1节点执行—创建工作目录并进入

mkdir -p ~/TLS/{etcd,k8s} && cd TLS/etcd

在k8s-master1节点执行—自签CA:期限10年,87600小时等于10年

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

在k8s-master1节点执行—创建ca-csr.json

cat > ca-csr.json << EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

在k8s-master1节点执行—生成证书

[root@k8s-master1 ~/TLS/etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
显示如下:
2020/11/18 15:23:03 [INFO] generating a new CA key and certificate from CSR
2020/11/18 15:23:03 [INFO] generate received request
2020/11/18 15:23:03 [INFO] received CSR
2020/11/18 15:23:03 [INFO] generating key: rsa-2048
2020/11/18 15:23:04 [INFO] encoded CSR
2020/11/18 15:23:04 [INFO] signed certificate with serial number 502216175606304420183686060059503154498973254382

在k8s-master1节点执行—查看证书

ls *pem
显示如下:
ca-key.pem  ca.pem

4. 使用自签CA签发Etcd HTTPS证书

在k8s-master1节点执行—创建证书申请文件:

cat > /root/TLS/etcd/server-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
    "42.51.80.131",
    "42.51.80.132",
    "42.51.80.133"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

注:上述文件hosts字段中IP为所有etcd节点的集群内部通信IP,一个都不能少!为了方便后期扩容可以多写几个预留的IP。

在k8s-master1节点执行—生成证书

[root@k8s-master1 ~/TLS/etcd]#cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
显示如下:
2020/11/18 15:24:26 [INFO] generate received request
2020/11/18 15:24:26 [INFO] received CSR
2020/11/18 15:24:26 [INFO] generating key: rsa-2048
2020/11/18 15:24:26 [INFO] encoded CSR
2020/11/18 15:24:26 [INFO] signed certificate with serial number 724406594509788710386266824287049394398073593453
2020/11/18 15:24:26 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

在k8s-master1节点执行—查看证书

ls server*pem
显示如下:
server-key.pem  server.pem

5. 部署Etcd集群

Github下载二进制文件
下载地址:
https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

以下在k8s-master1节点上操作,稍后将k8s-master1生成的所有文件拷贝到k8s-master2和k8s-master3

将etcd包上传至k8s-master1节点上的root目录下

[root@k8s-master1 ~]#ls /root/etcd-*
/root/etcd-v3.4.9-linux-amd64.tar.gz

在k8s-master1节点执行—然后创建工作目录并解压二进制包并移动两个文件到指定目录下

mkdir -p /opt/etcd/{bin,cfg,ssl}
tar zxf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
执行过程如下:
[root@k8s-master1 ~]#mkdir -p /opt/etcd/{bin,cfg,ssl}
[root@k8s-master1 ~]#tar zxf etcd-v3.4.9-linux-amd64.tar.gz
[root@k8s-master1 ~]#mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

在k8s-master1节点执行—创建etcd配置文件

cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://42.51.80.131:2380"
ETCD_LISTEN_CLIENT_URLS="https://42.51.80.131:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://42.51.80.131:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://42.51.80.131:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://42.51.80.131:2380,etcd-2=https://42.51.80.132:2380,etcd-3=https://42.51.80.133:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

参数解析:

ETCD_NAME:节点名称,集群中唯一
ETCD_DATA_DIR:数据目录
ETCD_LISTEN_PEER_URLS:集群通信监听地址
ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址
ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址
ETCD_INITIAL_CLUSTER:集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN:集群Token
ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new是新集群,existing表示加入已有集群

在k8s-master1节点执行—创建etcd服务,systemd管理etcd:

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

在k8s-master1节点执行—把刚才生成的证书拷贝到配置文件中的路径

cp -f ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/

在k8s-master1节点执行—将上面k8s-master1所有生成的文件拷贝到k8s-master2和k8s-master3

scp -r /opt/etcd/ root@k8s-master2:/opt/
scp -r /usr/lib/systemd/system/etcd.service root@k8s-master2:/usr/lib/systemd/system/
scp -r /opt/etcd/ root@k8s-master3:/opt/
scp -r /usr/lib/systemd/system/etcd.service root@k8s-master3:/usr/lib/systemd/system/
显示如下:
[root@k8s-master1 ~]#scp -r /opt/etcd/ root@k8s-master2:/opt/
etcd                                                                                                                                                                                                                                        100%   23MB 122.2MB/s   00:00    
etcdctl                                                                                                                                                                                                                                     100%   17MB 114.2MB/s   00:00    
etcd.conf                                                                                                                                                                                                                                   100%  501   823.4KB/s   00:00    
ca-key.pem                                                                                                                                                                                                                                  100% 1675     2.1MB/s   00:00    
ca.pem                                                                                                                                                                                                                                      100% 1265     2.3MB/s   00:00    
server-key.pem                                                                                                                                                                                                                              100% 1679     3.4MB/s   00:00    
server.pem                                                                                                                                                                                                                                  100% 1338     2.8MB/s   00:00    
[root@k8s-master1 ~]#scp -r /usr/lib/systemd/system/etcd.service root@k8s-master2:/usr/lib/systemd/system/
etcd.service                                                                                                                                                                                                                                100%  533   679.6KB/s   00:00    
[root@k8s-master1 ~]#scp -r /opt/etcd/ root@k8s-master3:/opt/
etcd                                                                                                                                                                                                                                        100%   23MB 122.1MB/s   00:00    
etcdctl                                                                                                                                                                                                                                     100%   17MB 105.3MB/s   00:00    
etcd.conf                                                                                                                                                                                                                                   100%  501   765.0KB/s   00:00    
ca-key.pem                                                                                                                                                                                                                                  100% 1675     3.1MB/s   00:00    
ca.pem                                                                                                                                                                                                                                      100% 1265     2.8MB/s   00:00    
server-key.pem                                                                                                                                                                                                                              100% 1679     3.3MB/s   00:00    
server.pem                                                                                                                                                                                                                                  100% 1338     2.9MB/s   00:00    
[root@k8s-master1 ~]#scp -r /usr/lib/systemd/system/etcd.service root@k8s-master3:/usr/lib/systemd/system/
etcd.service                                                                                                                                                                                                                                100%  533   776.5KB/s   00:00    

在k8s-master2和k8s-master3节点分别修改etcd.conf配置文件中的节点名称和当前服务器IP

vim /opt/etcd/cfg/etcd.conf

#[Member]
ETCD_NAME="etcd-x"                                              #修改此处,节点2改为etcd-2,节点3改为etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://42.51.80.xxx:2380"               #修改此处为当前服务器IP
ETCD_LISTEN_CLIENT_URLS="https://42.51.80.xxx:2379"             #修改此处为当前服务器IP
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://42.51.80.xxx:2380"    #修改此处为当前服务器IP
ETCD_ADVERTISE_CLIENT_URLS="https://42.51.80.xxx:2379"          #修改此处为当前服务器IP
ETCD_INITIAL_CLUSTER="etcd-1=https://42.51.80.131:2380,etcd-2=https://42.51.80.132:2380,etcd-3=https://42.51.80.133:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

同时启动并设置开机启动,k8s-master1,k8s-master2,k8s-master3

systemctl daemon-reload && systemctl enable etcd --now && systemctl status etcd

在任何master节点执行—查看集群状态

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://42.51.80.131:2379,https://42.51.80.132:2379,https://42.51.80.133:2379" endpoint health

输出结果如下

https://42.51.80.131:2379 is healthy: successfully committed proposal: took = 17.735641ms
https://42.51.80.132:2379 is healthy: successfully committed proposal: took = 16.272525ms
https://42.51.80.133:2379 is healthy: successfully committed proposal: took = 18.242957ms

如果输出上面信息,就说明集群部署成功。
如果有问题,第一步先看日志:/var/log/message 或 journalctl -u etcd

原文出处:myit -> https://myit.icu/index.php/archives/1033/

打赏

如果此文对你有所帮助,请随意打赏鼓励作者^_^