Kubernetes HA構成の作成 (1) etcd マルチノード TLS構成

以前HA構成の作り方を公開したが、kubeadmの公式に沿ってノードを作るため、再度記載。
この記事はetcdマルチノードのTLS構成をつくったときのメモ。

環境

etcd インストール

etcdのバージョンはkubernetes1.11と互換性があると書いてある3.2.18を用いる。
互換性についてはリリースノートを参照した。

本手順では、master上にetcdクラスタを構成する予定のため、master上で作業を行う。

etcdのダウンロード

ここを参考にetcdのバイナリをダウンロードする。
以下の作業は全てのmasterで行う。

$ ETCD_VER=v3.2.18
$ GOOGLE_URL=https://storage.googleapis.com/etcd
$ DOWNLOAD_URL=${GOOGLE_URL}
$ rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test
$ curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
$ tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1
$ rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

続いて、etcdとetcdctlをPATHの通っているところへ移動する。

$ sudo mv /tmp/etcd-download-test/etcd /usr/local/bin/
$ sudo mv /tmp/etcd-download-test/etcdctl /usr/local/bin/

etcdのTLSをセットアップ

TLSをセットアップ。このを参考に。
今回作成する証明書は3つ。
– クライアント証明書
– サーバがクライアントを認証するために使うもの。例としては、etcdctl
– サーバ証明書
– クライアントにてサーバが正しいか確認するよう使われる。
– ピア証明書
– etcdのメンバ間で使われる証明書

以下の作業はmaster1のみで行う。

cfsslをダウンロード

証明書作成のためcfsslをダウンロード。

curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o cfssl
chmod +x cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o cfssljson
chmod +x cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o cfssl-certinfo
chmod +x cfssl-certinfo
sudo mv cfssl cfssl-certinfo cfssljson /usr/local/bin/

CAの作成

必要なコンフィグの作成。

$ mkdir ~/cfssl
$ cd ~/cfssl
$ cfssl print-defaults config > ca-config.json
$ cfssl print-defaults csr > ca-csr.json

作成したテンプレートをベースに今回の環境用の設定を作成する

• ca-config.json

{
    "signing": {
        "default": {
            "expiry": "43800h"
        },
        "profiles": {
            "server": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

• ca-csr.json

{
    "CN": "test.local",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "JP",
            "L": "Tokyo",
            "ST": "Ota-ku"
        }
    ]
}

ここからCAを作成

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2018/09/12 09:32:29 [INFO] generating a new CA key and certificate from CSR
2018/09/12 09:32:29 [INFO] generate received request
2018/09/12 09:32:29 [INFO] received CSR
2018/09/12 09:32:29 [INFO] generating key: rsa-2048
2018/09/12 09:32:30 [INFO] encoded CSR
2018/09/12 09:32:30 [INFO] signed certificate with serial number 195402036152342129159335312985502183338061465596

正しく作成できると新しく以下のファイルができている

ca.csr
ca-key.pem
ca.pem

サーバ証明書の作成

先程と同様にテンプレートを作成

cfssl print-defaults csr > server.json

全てのmasterをhost内に含めるよう変更
今回は、全ホストで同じサーバ証明書を用いる。

• server.json

{
    "CN": "etcd",
    "hosts": [
        "192.168.110.246",
        "k8s-master1",
        "192.168.110.248",
        "k8s-master2",
        "192.168.110.244",
        "k8s-master3"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "JP",
            "L": "Tokyo",
            "ST": "Ota-ku"
        }
    ]
}

以下のコマンドで証明書の作成

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfsslj
son -bare server
2018/09/12 09:57:53 [INFO] generate received request
2018/09/12 09:57:53 [INFO] received CSR
2018/09/12 09:57:53 [INFO] generating key: rsa-2048
2018/09/12 09:57:53 [INFO] encoded CSR
2018/09/12 09:57:53 [INFO] signed certificate with serial number 470970637626621357035573631309534964329132128500
2018/09/12 09:57:53 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

正しく終了すると、以下のファイルができている。

server.csr
server-key.pem
server.pem

ピア証明書の作成

同様にテンプレートをつくって、各master用に作成。

$ cfssl print-defaults csr > k8s-master1.json

$ cp k8s-master1.json k8s-master2.json

$ cp k8s-master1.json k8s-master3.json

• k8s-master1

{
    "CN": "k8s-master1",
    "hosts": [
        "192.168.110.246",
        "k8s-master1"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "JP",
            "L": "Tokyo",
            "ST": "Ota-ku"
        }
    ]
}

k8s-master2,k8s-master3用には適宜IPとhostnameを変更する。

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer k8s-master1.json | cfssljson -bare k8s-master1
2018/09/12 11:41:26 [INFO] generate received request
2018/09/12 11:41:26 [INFO] received CSR
2018/09/12 11:41:26 [INFO] generating key: rsa-2048
2018/09/12 11:41:26 [INFO] encoded CSR
2018/09/12 11:41:26 [INFO] signed certificate with serial number 116626445458422141846757704040266452655628155548
2018/09/12 11:41:26 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer k8s-master2.json | cfssljson -bare k8s-master2
2018/09/12 11:41:36 [INFO] generate received request
2018/09/12 11:41:36 [INFO] received CSR
2018/09/12 11:41:36 [INFO] generating key: rsa-2048
2018/09/12 11:41:36 [INFO] encoded CSR
2018/09/12 11:41:36 [INFO] signed certificate with serial number 627454659134980543460959334257434685217817968879
2018/09/12 11:41:36 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer k8s-master3.json | cfssljson -bare k8s-master3
2018/09/12 11:41:47 [INFO] generate received request
2018/09/12 11:41:47 [INFO] received CSR
2018/09/12 11:41:47 [INFO] generating key: rsa-2048
2018/09/12 11:41:47 [INFO] encoded CSR
2018/09/12 11:41:47 [INFO] signed certificate with serial number 374921657428435121280556896769040191118517422115
2018/09/12 11:41:47 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

環境すると以下のファイルが作成される。

k8s-master1.csr
k8s-master1-key.pem
k8s-master1.pem
k8s-master2.csr
k8s-master2-key.pem
k8s-master2.pem
k8s-master3.csr
k8s-master3-key.pem
k8s-master3.pem

クライアント証明書の作成

最後にクライアント証明書の作成を行う。
テンプレートの作成

$ cfssl print-defaults csr > client.json

そして、参考ページ通りに変更を行う。hostsは空欄で作成。

• client.json

{
    "CN": "client",
    "hosts": [""],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "JP",
            "L": "Tokyo",
            "ST": "Ota-ku"
        }
    ]
}

証明書を同様に以下のコマンドで作成。

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare client
2018/09/12 11:51:24 [INFO] generate received request
2018/09/12 11:51:24 [INFO] received CSR
2018/09/12 11:51:24 [INFO] generating key: rsa-2048
2018/09/12 11:51:24 [INFO] encoded CSR
2018/09/12 11:51:24 [INFO] signed certificate with serial number 192269045916337779375023926646032559746885504728
2018/09/12 11:51:24 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
localadmin@k8s-master1:~/cfssl$

正しく完了すると、以下のファイルが作成される。

client.csr
client-key.pem
client.pem

各証明書を配布

各ノード内に作成した証明書を配置する。
証明書を置くフォルダを/etc/etcd/sslとする

sudo mkdir -p /etc/etcd/ssl

各ノードの証明書は以下の通りになる。

$ ls -1 /etc/etcd/ssl/
ca.pem
k8s-master1-key.pem
k8s-master1.pem
server-key.pem
server.pem

$ ls -1 /etc/etcd/ssl/
ca.pem
k8s-master2-key.pem
k8s-master2.pem
server-key.pem
server.pem

$ ls -1 /etc/etcd/ssl/
ca.pem
k8s-master3-key.pem
k8s-master3.pem
server-key.pem
server.pem

etcdのsystemdをセットアップ

etcdのデータフォルダとして/var/lib/etcdを指定するため、作成しておく。

$ sudo mkdir -p /var/lib/etcd
$ sudo chown -R root:$(whoami) /var/lib/etcd
$ sudo chmod -R a+rw /var/lib/etcd

各ノードのsystemdファイルを作成する。

以下はk8s-master1の/etc/systemd/system/etcd.serviceファイルになる。
hostnameやIPアドレスを変更して同様のファイルをk8s-master2,k8s-amster3にも作成する。

[Unit]
Description=Etcd Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
EnvironmentFile=-/etc/default/etcd
ExecStart=/usr/local/bin/etcd \
    --name k8s-master1 \
    --data-dir /var/lib/etcd/ \
    --listen-client-urls https://192.168.110.246:2379,https://127.0.0.1:2379 \
    --advertise-client-urls https://192.168.110.246:2379 \
    --listen-peer-urls https://192.168.110.246:2380 \
    --initial-advertise-peer-urls https://192.168.110.246:2380 \
    --cert-file=/etc/etcd/ssl/server.pem \
    --key-file=/etc/etcd/ssl/server-key.pem \
    --client-cert-auth \
    --trusted-ca-file=/etc/etcd/ssl/ca.pem \
    --peer-cert-file=/etc/etcd/ssl/k8s-master1.pem \
    --peer-key-file=/etc/etcd/ssl/k8s-master1-key.pem \
    --peer-client-cert-auth \
    --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
    --initial-cluster k8s-master1=https://192.168.110.246:2380,k8s-master2=https://192.168.110.248:2380,k8s-master3=https://192.168.110.244:2380 \
    --initial-cluster-state=new

Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

その後etcdを起動

systemctl daemon-reload
systemctl enable etcd.service
systemctl start etcd.service

etcdがactiveになっているか確認

$ sudo systemctl status etcd.service
● etcd.service - Etcd Server
   Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-09-12 12:18:17 +08; 2min 8s ago
 Main PID: 2300 (etcd)
    Tasks: 14
   Memory: 21.9M
      CPU: 2.685s
   CGroup: /system.slice/etcd.service
           └─2300 /usr/local/bin/etcd --name k8s-master1 --data-dir /var/lib/etcd/ --listen-client-urls https://192.168.110.246:2379,https://127.0.0.1:2379 --advertise-client-urls https://192.168.110.246:2379 --listen-peer-urls https://192.168.110.246:2380 --initial-adver

Sep 12 12:18:37 k8s-master1 etcd[2300]: WARNING: 2018/09/12 12:18:37 Failed to dial 192.168.110.246:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
Sep 12 12:18:37 k8s-master1 etcd[2300]: WARNING: 2018/09/12 12:18:37 Failed to dial 127.0.0.1:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
Sep 12 12:18:37 k8s-master1 etcd[2300]: health check for peer d3774d58a4e45bc8 could not connect: dial tcp 192.168.110.244:2380: getsockopt: connection refused
Sep 12 12:18:37 k8s-master1 etcd[2300]: health check for peer da06d0c251127a09 could not connect: dial tcp 192.168.110.248:2380: getsockopt: connection refused
Sep 12 12:18:42 k8s-master1 etcd[2300]: health check for peer d3774d58a4e45bc8 could not connect: dial tcp 192.168.110.244:2380: getsockopt: connection refused
Sep 12 12:18:45 k8s-master1 etcd[2300]: peer d3774d58a4e45bc8 became active
Sep 12 12:18:45 k8s-master1 etcd[2300]: established a TCP streaming connection with peer d3774d58a4e45bc8 (stream Message reader)
Sep 12 12:18:45 k8s-master1 etcd[2300]: established a TCP streaming connection with peer d3774d58a4e45bc8 (stream MsgApp v2 reader)
Sep 12 12:18:45 k8s-master1 etcd[2300]: established a TCP streaming connection with peer d3774d58a4e45bc8 (stream MsgApp v2 writer)
Sep 12 12:18:45 k8s-master1 etcd[2300]: established a TCP streaming connection with peer d3774d58a4e45bc8 (stream Message writer)

etcdの正常性確認

etcdの正常性を確認する。
先程の証明書を作ったノードから確認。
まずは、メンバーとクラスタの正常性を確認する

localadmin@k8s-master1:~$ sudo etcdctl --cert-file=/home/localadmin/cfssl/k8s-master1.pem --ca-file=/etc/etcd/ssl/ca.pem  --key-file=/home/localadmin/cfssl/k8s-master1-key.pem --endpoints=https://192.168.110.246:2379 member list
96d56b68d62db8da: name=k8s-master1 peerURLs=https://192.168.110.246:2380 clientURLs=https://192.168.110.246:2379 isLeader=false
d3774d58a4e45bc8: name=k8s-master3 peerURLs=https://192.168.110.244:2380 clientURLs=https://192.168.110.244:2379 isLeader=false
da06d0c251127a09: name=k8s-master2 peerURLs=https://192.168.110.248:2380 clientURLs=https://192.168.110.248:2379 isLeader=true

localadmin@k8s-master1:~$ sudo etcdctl --cert-file=/home/localadmin/cfssl/k8s-master1.pem --ca-file=/etc/etcd/ssl/ca.pem  --key-file=/home/localadmin/cfssl/k8s-master1-key.pem --endpoints=https://192.168.110.246:2379 cluster-health
member 96d56b68d62db8da is healthy: got healthy result from https://192.168.110.246:2379
member d3774d58a4e45bc8 is healthy: got healthy result from https://192.168.110.244:2379
member da06d0c251127a09 is healthy: got healthy result from https://192.168.110.248:2379
cluster is healthy

localadmin@k8s-master1:~$ sudo etcdctl --cert-file=/home/localadmin/cfssl/k8s-master2.pem --ca-file=/etc/etcd/ssl/ca.pem  --key-file=/home/localadmin/cfssl/k8s-master2-key.pem --endpoints=https://192.168.110.248:2379 cluster-health
member 96d56b68d62db8da is healthy: got healthy result from https://192.168.110.246:2379
member d3774d58a4e45bc8 is healthy: got healthy result from https://192.168.110.244:2379
member da06d0c251127a09 is healthy: got healthy result from https://192.168.110.248:2379
cluster is healthy

localadmin@k8s-master1:~$ sudo etcdctl --cert-file=/home/localadmin/cfssl/k8s-master3.pem --ca-file=/etc/etcd/ssl/ca.pem  --key-file=/home/localadmin/cfssl/k8s-master3-key.pem --endpoints=https://192.168.110.244:2379 cluster-health
member 96d56b68d62db8da is healthy: got healthy result from https://192.168.110.246:2379
member d3774d58a4e45bc8 is healthy: got healthy result from https://192.168.110.244:2379
member da06d0c251127a09 is healthy: got healthy result from https://192.168.110.248:2379
cluster is healthy

続いて、クライアント証明書を使用して、Create/Read/Deleteを試す。

localadmin@k8s-master1:~/cfssl$ etcdctl --ca-file=/etc/etcd/ssl/ca.pem --endpoints=https://192.168.110.248:2379,https://192.168.110.246:2379,https://192.168.110.244:2379 --cert-file ./client.pem --key-file ./client-key.pem set foo bar
bar
localadmin@k8s-master1:~/cfssl$ etcdctl  --ca-file=/etc/etcd/ssl/ca.pem --endpoints=https://192.168.110.248:2379,https://192.168.110.246:2379,https://192.168.110.244:2379 --cert-file ./client.pem --key-file ./client-key.pem get foo
bar
localadmin@k8s-master1:~/cfssl$ etcdctl  --ca-file=/etc/etcd/ssl/ca.pem --endpoints=https://192.168.110.248:2379,https://192.168.110.246:2379,https://192.168.110.244:2379 --cert-file ./client.pem --key-file ./client-key.pem rm foo
PrevNode.Value: bar

上記のように動作すれば問題なく動いている。