helmでIngress Controllerを導入する
Ingressを使ってアプリケーションを外部からアクセスできるために
NginxベースのIngress Controllerを導入。
目次
環境
今回の環境は以下の通り
$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master Ready master 9h v1.11.2 192.168.120.70 <none> Ubuntu 16.04.3 LTS 4.4.0-87-generic docker://17.3.2
k8s-node1 Ready <none> 9h v1.11.2 192.168.120.71 <none> Ubuntu 16.04.3 LTS 4.4.0-87-generic docker://17.3.2
k8s-node2 Ready <none> 9h v1.11.2 192.168.120.72 <none> Ubuntu 16.04.3 LTS 4.4.0-87-generic docker://17.3.2
k8s-node3 Ready <none> 9h v1.11.2 192.168.120.73 <none> Ubuntu 16.04.3 LTS 4.4.0-87-generic docker://17.3.2
helmの導入は完了済み。
$ kubectl -n kube-system get pod
NAME READY STATUS RESTARTS AGE
canal-5vcfw 3/3 Running 0 9h
canal-6hm4k 3/3 Running 0 9h
canal-hzxwm 3/3 Running 0 9h
canal-qz4zg 3/3 Running 0 9h
coredns-78fcdf6894-snc69 1/1 Running 0 9h
coredns-78fcdf6894-t8qsn 1/1 Running 0 9h
etcd-k8s-master 1/1 Running 0 9h
kube-apiserver-k8s-master 1/1 Running 0 9h
kube-controller-manager-k8s-master 1/1 Running 0 9h
kube-proxy-4dk2q 1/1 Running 0 9h
kube-proxy-trnft 1/1 Running 0 9h
kube-proxy-v5pvd 1/1 Running 0 9h
kube-proxy-vbhr5 1/1 Running 0 9h
kube-scheduler-k8s-master 1/1 Running 0 9h
tiller-deploy-56c4cf647b-r9vkf 1/1 Running 0 9h
helmでIngress Controllerの導入
Ingress Controllerをkube-systemネームスペース内の導入する。
使用するのは、以下のchart。
$ helm search nginx-ingress
NAME CHART VERSION APP VERSION DESCRIPTION
stable/nginx-ingress 0.28.2 0.19.0 An nginx Ingress controller that uses ConfigMap to store ...
ネームスペースとリリース名を指定してインストール。
$ helm install --name nginx-ingresscontroller --namespace kube-system stable/nginx-ingress
NAME: nginx-ingresscontroller
LAST DEPLOYED: Mon Sep 10 08:28:56 2018
NAMESPACE: kube-system
STATUS: DEPLOYED
RESOURCES:
==> v1beta1/Role
NAME AGE
nginx-ingresscontroller 0s
==> v1beta1/RoleBinding
NAME AGE
nginx-ingresscontroller 0s
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
nginx-ingresscontroller-controller 1 1 1 0 0s
nginx-ingresscontroller-default-backend 1 1 1 0 0s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
nginx-ingresscontroller-controller-cdd745574-6xmcv 0/1 ContainerCreating 0 0s
nginx-ingresscontroller-default-backend-67cf46b849-4vnlt 0/1 ContainerCreating 0 0s
==> v1/ConfigMap
NAME DATA AGE
nginx-ingresscontroller-controller 1 0s
==> v1/ServiceAccount
NAME SECRETS AGE
nginx-ingresscontroller 1 0s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingresscontroller-controller LoadBalancer 10.101.88.232 <pending> 80:31420/TCP,443:31091/TCP 0s
nginx-ingresscontroller-default-backend ClusterIP 10.108.73.1 <none> 80/TCP 0s
==> v1beta1/ClusterRole
NAME AGE
nginx-ingresscontroller 0s
==> v1beta1/ClusterRoleBinding
NAME AGE
nginx-ingresscontroller 0s
NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace kube-system get services -o wide -w nginx-ingresscontroller-controller'
An example Ingress that makes use of the controller:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
name: example
namespace: foo
spec:
rules:
- host: www.example.com
http:
paths:
- backend:
serviceName: exampleService
servicePort: 80
path: /
# This section is only required if TLS is to be enabled for the Ingress
tls:
- hosts:
- www.example.com
secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt: <base64 encoded cert>
tls.key: <base64 encoded key>
type: kubernetes.io/tls
ステータスを確認する。
この環境では、External-IPは割り当てられないので
各PodがReadyになったらNginx Ingress Controllerにアクセスして試す。
$ helm status nginx-ingresscontroller
LAST DEPLOYED: Mon Sep 10 08:28:56 2018
NAMESPACE: kube-system
STATUS: DEPLOYED
RESOURCES:
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingresscontroller-controller LoadBalancer 10.101.88.232 <pending> 80:31420/TCP,443:31091/TCP 2m
nginx-ingresscontroller-default-backend ClusterIP 10.108.73.1 <none> 80/TCP 2m
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
nginx-ingresscontroller-controller 1 1 1 1 2m
nginx-ingresscontroller-default-backend 1 1 1 1 2m
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
nginx-ingresscontroller-controller-cdd745574-6xmcv 1/1 Running 0 2m
nginx-ingresscontroller-default-backend-67cf46b849-4vnlt 1/1 Running 0 2m
==> v1/ServiceAccount
NAME SECRETS AGE
nginx-ingresscontroller 1 2m
==> v1beta1/ClusterRole
NAME AGE
nginx-ingresscontroller 2m
==> v1beta1/Role
nginx-ingresscontroller 2m
==> v1beta1/RoleBinding
NAME AGE
nginx-ingresscontroller 2m
==> v1/ConfigMap
NAME DATA AGE
nginx-ingresscontroller-controller 1 2m
==> v1beta1/ClusterRoleBinding
NAME AGE
nginx-ingresscontroller 2m
<snip>
IngressControllerにアクセスする
まず、割り当てられたNodePortを確認する。
$ kubectl get svc nginx-ingresscontroller-controller -o yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2018-09-10T00:28:57Z
labels:
app: nginx-ingress
chart: nginx-ingress-0.28.2
component: controller
heritage: Tiller
release: nginx-ingresscontroller
name: nginx-ingresscontroller-controller
namespace: kube-system
resourceVersion: "57777"
selfLink: /api/v1/namespaces/kube-system/services/nginx-ingresscontroller-controller
uid: 813b319c-b490-11e8-be54-005056a3e36a
spec:
clusterIP: 10.101.88.232
externalTrafficPolicy: Cluster
ports:
- name: http
nodePort: 31420
port: 80
protocol: TCP
targetPort: http
- name: https
nodePort: 31091
port: 443
protocol: TCP
targetPort: https
selector:
app: nginx-ingress
component: controller
release: nginx-ingresscontroller
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer: {}
http用のポートは31420なので、k8s-nodeのIPアドレスとポート番号でnginxのIngress Controllerにアクセスできる。
ちなみに、どのノードのIPを使おうがnginxにルーティングされる。
理由はiptablesでNATしているから。以下がテーブルを追ったときのログ。
$ sudo iptables -t nat -4 -L PREROUTING
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
cali-PREROUTING all -- anywhere anywhere /* cali:6gwbT8clXdHdC1b1 */
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
$ sudo iptables -t nat -4 -L KUBE-SERVICES
Chain KUBE-SERVICES (2 references)
target prot opt source destination
KUBE-MARK-MASQ udp -- !10.244.0.0/16 10.96.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
KUBE-SVC-TCOU7JCQXEZGVUNU udp -- anywhere 10.96.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.96.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- anywhere 10.96.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.96.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- anywhere 10.96.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.109.238.136 /* kube-system/tiller-deploy:tiller cluster IP */ tcp dpt:44134
KUBE-SVC-K7J76NXP7AUZVFGS tcp -- anywhere 10.109.238.136 /* kube-system/tiller-deploy:tiller cluster IP */ tcp dpt:44134
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.101.88.232 /* kube-system/nginx-ingresscontroller-controller:http cluster IP */ tcp dpt:http
KUBE-SVC-QPYN3OELPNPAWLKR tcp -- anywhere 10.101.88.232 /* kube-system/nginx-ingresscontroller-controller:http cluster IP */ tcp dpt:http
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.101.88.232 /* kube-system/nginx-ingresscontroller-controller:https cluster IP */ tcp dpt:https
KUBE-SVC-NBBB5YYR4PDL3QKB tcp -- anywhere 10.101.88.232 /* kube-system/nginx-ingresscontroller-controller:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.108.73.1 /* kube-system/nginx-ingresscontroller-default-backend:http cluster IP */ tcp dpt:http
KUBE-SVC-S6BQWJKA2XE4MCZM tcp -- anywhere 10.108.73.1 /* kube-system/nginx-ingresscontroller-default-backend:http cluster IP */ tcp dpt:http
KUBE-NODEPORTS all -- anywhere anywhere /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
$ sudo iptables -t nat -4 -L KUBE-NODEPORTS
Chain KUBE-NODEPORTS (1 references)
target prot opt source destination
KUBE-MARK-MASQ tcp -- anywhere anywhere /* kube-system/nginx-ingresscontroller-controller:http */ tcp dpt:31420
KUBE-SVC-QPYN3OELPNPAWLKR tcp -- anywhere anywhere /* kube-system/nginx-ingresscontroller-controller:http */ tcp dpt:31420
KUBE-MARK-MASQ tcp -- anywhere anywhere /* kube-system/nginx-ingresscontroller-controller:https */ tcp dpt:31091
KUBE-SVC-NBBB5YYR4PDL3QKB tcp -- anywhere anywhere /* kube-system/nginx-ingresscontroller-controller:https */ tcp dpt:31091
$ sudo iptables -t nat -4 -L KUBE-MARK-MASQ
Chain KUBE-MARK-MASQ (18 references)
target prot opt source destination
MARK all -- anywhere anywhere MARK or 0x4000
$ sudo iptables -t nat -4 -L KUBE-SVC-QPYN3OELPNPAWLKR
Chain KUBE-SVC-QPYN3OELPNPAWLKR (2 references)
target prot opt source destination
KUBE-SEP-M6Y5JS7QXZ2GKQMU all -- anywhere anywhere /* kube-system/nginx-ingresscontroller-controller:http */
$ sudo iptables -t nat -4 -L KUBE-SEP-M6Y5JS7QXZ2GKQMU
Chain KUBE-SEP-M6Y5JS7QXZ2GKQMU (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 10.244.2.3 anywhere /* kube-system/nginx-ingresscontroller-controller:http */
DNAT tcp -- anywhere anywhere /* kube-system/nginx-ingresscontroller-controller:http */ tcp to:10.244.2.3:80
$ sudo iptables -t nat -4 -L POSTROUTING
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
cali-POSTROUTING all -- anywhere anywhere /* cali:O3lYWMrLQYEMJtB5 */
KUBE-POSTROUTING all -- anywhere anywhere /* kubernetes postrouting rules */
MASQUERADE all -- 172.17.0.0/16 anywhere
RETURN all -- 10.244.0.0/16 10.244.0.0/16
MASQUERADE all -- 10.244.0.0/16 !base-address.mcast.net/4
RETURN all -- !10.244.0.0/16 10.244.0.0/24
MASQUERADE all -- !10.244.0.0/16 10.244.0.0/16
$ sudo iptables -t nat -4 -L KUBE-POSTROUTING
Chain KUBE-POSTROUTING (1 references)
target prot opt source destination
MASQUERADE all -- anywhere anywhere /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
PREROUTINGでDNATして、POSTROUTINGでMASQUERADEしているので、パケットフローは以下の図のようになる。
外部からとりあえずアクセスしてみると、デフォルトのバックエンドから404が返ってきて動いてそう
$ curl -v http://192.168.120.70:31420
* Rebuilt URL to: http://192.168.120.70:31420/
* Trying 192.168.120.70...
* Connected to 192.168.120.70 (192.168.120.70) port 31420 (#0)
> GET / HTTP/1.1
> Host: 192.168.120.70:31420
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: nginx/1.15.3
< Date: Mon, 10 Sep 2018 01:57:37 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 21
< Connection: keep-alive
<
* Connection #0 to host 192.168.120.70 left intact
default backend - 404
Ingressを作って動作確認
実際にIngressを作って動作確認を行う。
今回はtest.corp.localドメイン上に作成。
DNSサーバには*.test.corp.localがMasterのIPになるように設定しておく。
以下3つのyamlを使ってnginxをサービスごとたてる
$ kubectl create -f nginx-web-deployments.yaml
$ kubectl create -f nginx-web-service.yaml
$ kubectl create -f nginx-web-ingress.yaml
yamlファイルの中身は以下の通り。
# nginx-web-deployments.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-web-demo
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
role: web
spec:
containers:
- name: nginx-web
image: nginx
ports:
- containerPort: 80
--
# nginx-web-service.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx-demo-svc
labels:
app: nginx
role: web
spec:
ports:
# the port that this service should serve on
- port: 80
targetPort: 80
protocol: TCP
name: tcp
selector:
app: nginx
role: web
--
# nginx-web-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-demo-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: nginx-web.test.corp.local
http:
paths:
- path: "/"
backend:
serviceName: nginx-demo-svc
servicePort: 80
Ingressが作成できたらアクセスしてみる。
アクセスする際には、http://nginx-web.test.corp.local:31420のようにpathとnodeportを指定する必要がある。
ここにexternalIPが割り当てられて、80ポートを開放できればノードポートを指定する必要がない。
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
nginx-demo-ingress nginx-web.test.corp.local 80 18m
$ curl http://nginx-web.test.corp.local:31420
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
まとめ
helmを使ってnginx ingress controllerを作成した。
基本的なパッケージはhelmを使って作れるので、大変楽。
もし、変更を加えたい場合には、自作のchartを作らければいけないのだろうか？