HelmでNginx Ingress Controllerを導入

helmでIngress Controllerを導入する
Ingressを使ってアプリケーションを外部からアクセスできるために
NginxベースのIngress Controllerを導入。

環境

今回の環境は以下の通り

$ kubectl get node -o wide
NAME         STATUS    ROLES     AGE       VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
k8s-master   Ready     master    9h        v1.11.2   192.168.120.70   <none>        Ubuntu 16.04.3 LTS   4.4.0-87-generic   docker://17.3.2
k8s-node1    Ready     <none>    9h        v1.11.2   192.168.120.71   <none>        Ubuntu 16.04.3 LTS   4.4.0-87-generic   docker://17.3.2
k8s-node2    Ready     <none>    9h        v1.11.2   192.168.120.72   <none>        Ubuntu 16.04.3 LTS   4.4.0-87-generic   docker://17.3.2
k8s-node3    Ready     <none>    9h        v1.11.2   192.168.120.73   <none>        Ubuntu 16.04.3 LTS   4.4.0-87-generic   docker://17.3.2

helmの導入は完了済み。

$ kubectl -n kube-system get pod
NAME                                 READY     STATUS    RESTARTS   AGE
canal-5vcfw                          3/3       Running   0          9h
canal-6hm4k                          3/3       Running   0          9h
canal-hzxwm                          3/3       Running   0          9h
canal-qz4zg                          3/3       Running   0          9h
coredns-78fcdf6894-snc69             1/1       Running   0          9h
coredns-78fcdf6894-t8qsn             1/1       Running   0          9h
etcd-k8s-master                      1/1       Running   0          9h
kube-apiserver-k8s-master            1/1       Running   0          9h
kube-controller-manager-k8s-master   1/1       Running   0          9h
kube-proxy-4dk2q                     1/1       Running   0          9h
kube-proxy-trnft                     1/1       Running   0          9h
kube-proxy-v5pvd                     1/1       Running   0          9h
kube-proxy-vbhr5                     1/1       Running   0          9h
kube-scheduler-k8s-master            1/1       Running   0          9h
tiller-deploy-56c4cf647b-r9vkf       1/1       Running   0          9h

helmでIngress Controllerの導入

Ingress Controllerをkube-systemネームスペース内の導入する。
使用するのは、以下のchart。

$ helm search nginx-ingress
NAME                    CHART VERSION   APP VERSION DESCRIPTION
stable/nginx-ingress    0.28.2          0.19.0      An nginx Ingress controller that uses ConfigMap to store ...

ネームスペースとリリース名を指定してインストール。

$ helm install --name nginx-ingresscontroller --namespace kube-system stable/nginx-ingress
NAME:   nginx-ingresscontroller
LAST DEPLOYED: Mon Sep 10 08:28:56 2018
NAMESPACE: kube-system
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/Role
NAME                     AGE
nginx-ingresscontroller  0s

==> v1beta1/RoleBinding
NAME                     AGE
nginx-ingresscontroller  0s

==> v1beta1/Deployment
NAME                                     DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
nginx-ingresscontroller-controller       1        1        1           0          0s
nginx-ingresscontroller-default-backend  1        1        1           0          0s

==> v1/Pod(related)
NAME                                                      READY  STATUS             RESTARTS  AGE
nginx-ingresscontroller-controller-cdd745574-6xmcv        0/1    ContainerCreating  0         0s
nginx-ingresscontroller-default-backend-67cf46b849-4vnlt  0/1    ContainerCreating  0         0s

==> v1/ConfigMap
NAME                                DATA  AGE
nginx-ingresscontroller-controller  1     0s

==> v1/ServiceAccount
NAME                     SECRETS  AGE
nginx-ingresscontroller  1        0s

==> v1/Service
NAME                                     TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
nginx-ingresscontroller-controller       LoadBalancer  10.101.88.232  <pending>    80:31420/TCP,443:31091/TCP  0s
nginx-ingresscontroller-default-backend  ClusterIP     10.108.73.1    <none>       80/TCP                      0s

==> v1beta1/ClusterRole
NAME                     AGE
nginx-ingresscontroller  0s

==> v1beta1/ClusterRoleBinding
NAME                     AGE
nginx-ingresscontroller  0s


NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace kube-system get services -o wide -w nginx-ingresscontroller-controller'

An example Ingress that makes use of the controller:

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

ステータスを確認する。
この環境では、External-IPは割り当てられないのでで問題ない。
各PodがReadyになったらNginx Ingress Controllerにアクセスして試す。

$ helm status nginx-ingresscontroller
LAST DEPLOYED: Mon Sep 10 08:28:56 2018
NAMESPACE: kube-system
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                                     TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
nginx-ingresscontroller-controller       LoadBalancer  10.101.88.232  <pending>    80:31420/TCP,443:31091/TCP  2m
nginx-ingresscontroller-default-backend  ClusterIP     10.108.73.1    <none>       80/TCP                      2m

==> v1beta1/Deployment
NAME                                     DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
nginx-ingresscontroller-controller       1        1        1           1          2m
nginx-ingresscontroller-default-backend  1        1        1           1          2m

==> v1/Pod(related)
NAME                                                      READY  STATUS   RESTARTS  AGE
nginx-ingresscontroller-controller-cdd745574-6xmcv        1/1    Running  0         2m
nginx-ingresscontroller-default-backend-67cf46b849-4vnlt  1/1    Running  0         2m

==> v1/ServiceAccount
NAME                     SECRETS  AGE
nginx-ingresscontroller  1        2m

==> v1beta1/ClusterRole
NAME                     AGE
nginx-ingresscontroller  2m

==> v1beta1/Role
nginx-ingresscontroller  2m

==> v1beta1/RoleBinding
NAME                     AGE
nginx-ingresscontroller  2m

==> v1/ConfigMap
NAME                                DATA  AGE
nginx-ingresscontroller-controller  1     2m

==> v1beta1/ClusterRoleBinding
NAME                     AGE
nginx-ingresscontroller  2m

<snip>

IngressControllerにアクセスする

まず、割り当てられたNodePortを確認する。

$ kubectl get svc nginx-ingresscontroller-controller -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2018-09-10T00:28:57Z
  labels:
    app: nginx-ingress
    chart: nginx-ingress-0.28.2
    component: controller
    heritage: Tiller
    release: nginx-ingresscontroller
  name: nginx-ingresscontroller-controller
  namespace: kube-system
  resourceVersion: "57777"
  selfLink: /api/v1/namespaces/kube-system/services/nginx-ingresscontroller-controller
  uid: 813b319c-b490-11e8-be54-005056a3e36a
spec:
  clusterIP: 10.101.88.232
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    nodePort: 31420
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 31091
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: nginx-ingress
    component: controller
    release: nginx-ingresscontroller
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

http用のポートは31420なので、k8s-nodeのIPアドレスとポート番号でnginxのIngress Controllerにアクセスできる。
ちなみに、どのノードのIPを使おうがnginxにルーティングされる。
理由はiptablesでNATしているから。以下がテーブルを追ったときのログ。

$ sudo iptables -t nat -4 -L PREROUTING
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
cali-PREROUTING  all  --  anywhere             anywhere             /* cali:6gwbT8clXdHdC1b1 */
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

$ sudo iptables -t nat -4 -L KUBE-SERVICES
Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-MARK-MASQ  udp  -- !10.244.0.0/16        10.96.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  anywhere             10.96.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
KUBE-MARK-MASQ  tcp  -- !10.244.0.0/16        10.96.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  anywhere             10.96.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
KUBE-MARK-MASQ  tcp  -- !10.244.0.0/16        10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  anywhere             10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.244.0.0/16        10.109.238.136       /* kube-system/tiller-deploy:tiller cluster IP */ tcp dpt:44134
KUBE-SVC-K7J76NXP7AUZVFGS  tcp  --  anywhere             10.109.238.136       /* kube-system/tiller-deploy:tiller cluster IP */ tcp dpt:44134
KUBE-MARK-MASQ  tcp  -- !10.244.0.0/16        10.101.88.232        /* kube-system/nginx-ingresscontroller-controller:http cluster IP */ tcp dpt:http
KUBE-SVC-QPYN3OELPNPAWLKR  tcp  --  anywhere             10.101.88.232        /* kube-system/nginx-ingresscontroller-controller:http cluster IP */ tcp dpt:http
KUBE-MARK-MASQ  tcp  -- !10.244.0.0/16        10.101.88.232        /* kube-system/nginx-ingresscontroller-controller:https cluster IP */ tcp dpt:https
KUBE-SVC-NBBB5YYR4PDL3QKB  tcp  --  anywhere             10.101.88.232        /* kube-system/nginx-ingresscontroller-controller:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.244.0.0/16        10.108.73.1          /* kube-system/nginx-ingresscontroller-default-backend:http cluster IP */ tcp dpt:http
KUBE-SVC-S6BQWJKA2XE4MCZM  tcp  --  anywhere             10.108.73.1          /* kube-system/nginx-ingresscontroller-default-backend:http cluster IP */ tcp dpt:http
KUBE-NODEPORTS  all  --  anywhere             anywhere             /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

$ sudo iptables -t nat -4 -L KUBE-NODEPORTS
Chain KUBE-NODEPORTS (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  tcp  --  anywhere             anywhere             /* kube-system/nginx-ingresscontroller-controller:http */ tcp dpt:31420
KUBE-SVC-QPYN3OELPNPAWLKR  tcp  --  anywhere             anywhere             /* kube-system/nginx-ingresscontroller-controller:http */ tcp dpt:31420
KUBE-MARK-MASQ  tcp  --  anywhere             anywhere             /* kube-system/nginx-ingresscontroller-controller:https */ tcp dpt:31091
KUBE-SVC-NBBB5YYR4PDL3QKB  tcp  --  anywhere             anywhere             /* kube-system/nginx-ingresscontroller-controller:https */ tcp dpt:31091

$ sudo iptables -t nat -4 -L KUBE-MARK-MASQ
Chain KUBE-MARK-MASQ (18 references)
target     prot opt source               destination
MARK       all  --  anywhere             anywhere             MARK or 0x4000

$ sudo iptables -t nat -4 -L KUBE-SVC-QPYN3OELPNPAWLKR
Chain KUBE-SVC-QPYN3OELPNPAWLKR (2 references)
target     prot opt source               destination
KUBE-SEP-M6Y5JS7QXZ2GKQMU  all  --  anywhere             anywhere             /* kube-system/nginx-ingresscontroller-controller:http */

$ sudo iptables -t nat -4 -L KUBE-SEP-M6Y5JS7QXZ2GKQMU
Chain KUBE-SEP-M6Y5JS7QXZ2GKQMU (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.244.2.3           anywhere             /* kube-system/nginx-ingresscontroller-controller:http */
DNAT       tcp  --  anywhere             anywhere             /* kube-system/nginx-ingresscontroller-controller:http */ tcp to:10.244.2.3:80

$ sudo iptables -t nat -4 -L POSTROUTING
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
cali-POSTROUTING  all  --  anywhere             anywhere             /* cali:O3lYWMrLQYEMJtB5 */
KUBE-POSTROUTING  all  --  anywhere             anywhere             /* kubernetes postrouting rules */
MASQUERADE  all  --  172.17.0.0/16        anywhere
RETURN     all  --  10.244.0.0/16        10.244.0.0/16
MASQUERADE  all  --  10.244.0.0/16       !base-address.mcast.net/4
RETURN     all  -- !10.244.0.0/16        10.244.0.0/24
MASQUERADE  all  -- !10.244.0.0/16        10.244.0.0/16

$ sudo iptables -t nat -4 -L KUBE-POSTROUTING
Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination
MASQUERADE  all  --  anywhere             anywhere             /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

PREROUTINGでDNATして、POSTROUTINGでMASQUERADEしているので、パケットフローは以下の図のようになる。

Nodeport paket flow

外部からとりあえずアクセスしてみると、デフォルトのバックエンドから404が返ってきて動いてそう

$ curl -v http://192.168.120.70:31420
* Rebuilt URL to: http://192.168.120.70:31420/
*   Trying 192.168.120.70...
* Connected to 192.168.120.70 (192.168.120.70) port 31420 (#0)
> GET / HTTP/1.1
> Host: 192.168.120.70:31420
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: nginx/1.15.3
< Date: Mon, 10 Sep 2018 01:57:37 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 21
< Connection: keep-alive
<
* Connection #0 to host 192.168.120.70 left intact
default backend - 404

Ingressを作って動作確認

実際にIngressを作って動作確認を行う。
今回はtest.corp.localドメイン上に作成。
DNSサーバには*.test.corp.localがMasterのIPになるように設定しておく。

以下3つのyamlを使ってnginxをサービスごとたてる

$ kubectl create -f nginx-web-deployments.yaml
$ kubectl create -f nginx-web-service.yaml
$ kubectl create -f nginx-web-ingress.yaml

yamlファイルの中身は以下の通り。

# nginx-web-deployments.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-web-demo
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
        role: web
    spec:
      containers:
      - name: nginx-web
        image: nginx
        ports:
        - containerPort: 80
--
# nginx-web-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-demo-svc
  labels:
    app: nginx
    role: web
spec:
  ports:
    # the port that this service should serve on
    - port: 80
      targetPort: 80
      protocol: TCP
      name: tcp
  selector:
    app: nginx
    role: web
--
# nginx-web-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-demo-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: nginx-web.test.corp.local
    http:
      paths:
      - path: "/"
        backend:
          serviceName: nginx-demo-svc
          servicePort: 80

Ingressが作成できたらアクセスしてみる。
アクセスする際には、http://nginx-web.test.corp.local:31420のようにpathとnodeportを指定する必要がある。
ここにexternalIPが割り当てられて、80ポートを開放できればノードポートを指定する必要がない。

$ kubectl get ingress
NAME                 HOSTS                       ADDRESS   PORTS     AGE
nginx-demo-ingress   nginx-web.test.corp.local             80        18m
$ curl http://nginx-web.test.corp.local:31420
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

まとめ

helmを使ってnginx ingress controllerを作成した。
基本的なパッケージはhelmを使って作れるので、大変楽。
もし、変更を加えたい場合には、自作のchartを作らければいけないのだろうか?

コメントを残す

メールアドレスが公開されることはありません。 * が付いている欄は必須項目です

CAPTCHA


このサイトはスパムを低減するために Akismet を使っています。コメントデータの処理方法の詳細はこちらをご覧ください