Helmを使ってPrometheusをkubernetesにインストールしたPrometheusにサービスの監視などをさせてみようと思い立ったはいいものの、クラスタ外部からClusterIPやkube-dnsに外部からアクセスするのが難しく頓挫。
いっそ中に立ててしまえと思い、helmのChartにあったので使うことに。
というか、内部に作るのが正しいのかな？
環境
$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
test-master Ready <none> 9h v1.11.1 10.16.181.91 Ubuntu 18.04.1 LTS 4.15.0-30-generic docker://17.12.1-ce
test-node1 Ready <none> 14m v1.11.1 10.16.181.92 Ubuntu 18.04.1 LTS 4.15.0-20-generic docker://17.12.1-ce
test-node2 Ready <none> 8m v1.11.1 10.16.181.93 Ubuntu 18.04.1 LTS 4.15.0-20-generic docker://17.12.1-ce
helmでprometheusのインストール
$helm search prometheus
NAME CHART VERSION APP VERSION DESCRIPTION
stable/prometheus 6.7.2 2.2.1 Prometheus is a monitoring system and time seri...
<snip>
バージョン2.2.1のChartを発見。早速導入。
その前に、namespaceの作成して、default Namespaceをmonitoringに
$ kubectl create ns monitoring
namespace "monitoring" created
$ kubens monitoring
Context "test-k8s" modified.
Active namespace is "monitoring".
ちなみに、kubensはここのツール。
めちゃくちゃ便利で大変お世話に。
では、prometheusのインストール
$ helm install --name prometheus stable/prometheus
$ helm install --name prometheus stable/prometheus
NAME: prometheus
LAST DEPLOYED: Mon Aug 13 09:59:25 2018
NAMESPACE: monitoring
STATUS: DEPLOYED
RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
prometheus-alertmanager 1 4s
prometheus-server 3 4s
==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
prometheus-alertmanager Pending vsphere-sc 4s
prometheus-server Pending vsphere-sc 4s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
prometheus-alertmanager ClusterIP 10.96.119.58 <none> 80/TCP 3s
prometheus-kube-state-metrics ClusterIP None <none> 80/TCP 3s
prometheus-node-exporter ClusterIP None <none> 9100/TCP 3s
prometheus-pushgateway ClusterIP 10.100.136.122 <none> 9091/TCP 3s
prometheus-server ClusterIP 10.110.235.79 <none> 80/TCP 3s
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
prometheus-alertmanager 1 1 1 0 3s
prometheus-kube-state-metrics 1 1 1 0 3s
prometheus-pushgateway 1 1 1 0 3s
prometheus-server 1 1 1 0 3s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
prometheus-node-exporter-kns6v 0/1 ContainerCreating 0 2s
prometheus-node-exporter-l49wp 0/1 ContainerCreating 0 3s
prometheus-node-exporter-wqj94 0/1 ContainerCreating 0 2s
prometheus-alertmanager-6b8897ddfc-5kztr 0/2 Pending 0 2s
prometheus-kube-state-metrics-8668948654-2jkbn 0/1 ContainerCreating 0 2s
prometheus-pushgateway-ff5689968-l2prj 0/1 ContainerCreating 0 2s
prometheus-server-78f564dfdb-hvvxk 0/2 Pending 0 2s
==> v1/ServiceAccount
NAME SECRETS AGE
prometheus-alertmanager 1 4s
prometheus-kube-state-metrics 1 4s
prometheus-node-exporter 1 4s
prometheus-pushgateway 1 4s
prometheus-server 1 3s
==> v1beta1/ClusterRole
NAME AGE
prometheus-kube-state-metrics 3s
prometheus-server 3s
==> v1beta1/ClusterRoleBinding
NAME AGE
prometheus-kube-state-metrics 3s
prometheus-server 3s
==> v1beta1/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
prometheus-node-exporter 3 3 0 3 0 <none> 3s
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.monitoring.svc.cluster.local
Get the Prometheus server URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 9090
The Prometheus alertmanager can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-alertmanager.monitoring.svc.cluster.local
Get the Alertmanager URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 9093
The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-pushgateway.monitoring.svc.cluster.local
Get the PushGateway URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 9091
For more information on running Prometheus, visit:
https://prometheus.io/
ちなみに、PersistentVolumeを使っているので、CloudProviderの設定が必要
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
prometheus-alertmanager Bound pvc-1faa9e0b-9e94-11e8-b2e2-005056a831ff 2Gi RWO vsphere-sc 1m
prometheus-server Bound pvc-1fab7c5a-9e94-11e8-b2e2-005056a831ff 8Gi RWO vsphere-sc 1m
prometheus用Ingressの作成
外からprometheusにアクセスできるようにprometheus用ingressを作成
# prometheus-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: promehteus-ingress
spec:
rules:
- host: prometheus.test.local
http:
paths:
- backend:
serviceName: prometheus-server
servicePort: 80
prometheus.test.localからMasterに対してDNSエントリを作成して、
外部からhttp://prometheus.test.local:<NodePort>/でアクセスすると見慣れた画面が見える。
Status -> Targetで正常に動作しているか確認。
これでprometheusのインストールは終了。
なんて簡単なんだ。。。