IsitoのTelemetryのデモを触ってみる

IsitoのTelemetryについてデモを使って確認する。

環境

公式のデモアプリケーションが動作済みを前提とする。
デモアプリケーションの作成方法はここを参照

$ kubectl get node -o wide
NAME          STATUS    ROLES     AGE       VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
k8s-master1   Ready     master    34d       v1.11.3   192.168.110.246   <none>        Ubuntu 16.04.5 LTS   4.4.0-72-generic   docker://17.3.2
k8s-master2   Ready     master    34d       v1.11.3   192.168.110.248   <none>        Ubuntu 16.04.5 LTS   4.4.0-72-generic   docker://17.3.2
k8s-master3   Ready     master    34d       v1.11.3   192.168.110.244   <none>        Ubuntu 16.04.5 LTS   4.4.0-72-generic   docker://17.3.2
k8s-node1     Ready     <none>    34d       v1.11.3   192.168.110.243   <none>        Ubuntu 16.04.5 LTS   4.4.0-72-generic   docker://17.3.2
k8s-node2     Ready     <none>    34d       v1.11.3   192.168.110.241   <none>        Ubuntu 16.04.5 LTS   4.4.0-72-generic   docker://17.3.2
k8s-node3     Ready     <none>    34d       v1.11.3   192.168.110.247   <none>        Ubuntu 16.04.5 LTS   4.4.0-72-generic   docker://17.3.2
$
$ kubectl -n istio-system get pod -o wide
NAME                                      READY     STATUS      RESTARTS   AGE       IP            NODE        NOMINATED NODE
istio-citadel-8545986b5c-qnpt6            1/1       Running     1          1d        10.244.5.62   k8s-node3   <none>
istio-cleanup-secrets-2srbf               0/1       Completed   0          1d        10.244.4.15   k8s-node2   <none>
istio-egressgateway-f449f844-d7mf6        1/1       Running     1          1d        10.244.4.22   k8s-node2   <none>
istio-galley-5c77f544cc-tl428             1/1       Running     1          1d        10.244.5.61   k8s-node3   <none>
istio-ingressgateway-7fc4cdbfc7-gdkpm     1/1       Running     1          1d        10.244.5.60   k8s-node3   <none>
istio-pilot-7974d67fcc-clhlt              2/2       Running     2          1d        10.244.3.23   k8s-node1   <none>
istio-policy-6bc68c8bcb-56zw2             2/2       Running     3          1d        10.244.5.63   k8s-node3   <none>
istio-security-post-install-8dppl         0/1       Completed   0          1d        10.244.3.20   k8s-node1   <none>
istio-sidecar-injector-74cd5c9565-cxc29   1/1       Running     1          1d        10.244.4.24   k8s-node2   <none>
istio-telemetry-7658f85655-5kpwb          2/2       Running     3          1d        10.244.4.20   k8s-node2   <none>
prometheus-554b5cc795-tbkjl               1/1       Running     1          1d        10.244.4.23   k8s-node2   <none>
$
$ kubectl -n istio-demo get pod -o wide
NAME                            READY     STATUS    RESTARTS   AGE       IP            NODE        NOMINATED NODE
details-v1-64f56d694d-znxpw     2/2       Running   0          23h       10.244.5.65   k8s-node3   <none>
productpage-v1-b8cf99cd-ghl92   2/2       Running   0          23h       10.244.5.67   k8s-node3   <none>
ratings-v1-8546767f5b-6vwrt     2/2       Running   0          23h       10.244.4.26   k8s-node2   <none>
reviews-v1-546d9f77ff-8mrmk     2/2       Running   0          23h       10.244.5.66   k8s-node3   <none>
reviews-v2-8fc76589f-4n5tl      2/2       Running   0          23h       10.244.4.27   k8s-node2   <none>
reviews-v3-7fdf5c754c-d82vc     2/2       Running   0          23h       10.244.3.25   k8s-node1   <none>

公式のデモ(メトリック)

公式のデモをベースに機能を確認する。

下記のメトリックを追加。
doublerequestcountは新しいメトリックの定義となり、reporter,source,destination,messageの情報をもち、valueはリクエストが来るたびに2加算される。
続いて、doublehandlerにてprometheusが先程のメトリックをどう処理するか記載し、ruleにてmetricとprometheusのhandlerを結びつける。

# Configuration for metric instances
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
  name: doublerequestcount
  namespace: istio-system
spec:
  value: "2" # count each request twice
  dimensions:
    reporter: conditional((context.reporter.kind | "inbound") == "outbound", "client", "server")
    source: source.workload.name | "unknown"
    destination: destination.workload.name | "unknown"
    message: '"twice the fun!"'
  monitored_resource_type: '"UNSPECIFIED"'
---
# Configuration for a Prometheus handler
apiVersion: "config.istio.io/v1alpha2"
kind: prometheus
metadata:
  name: doublehandler
  namespace: istio-system
spec:
  metrics:
  - name: double_request_count # Prometheus metric name
    instance_name: doublerequestcount.metric.istio-system # Mixer instance name (fully-qualified)
    kind: COUNTER
    label_names:
    - reporter
    - source
    - destination
    - message
---
# Rule to send metric instances to a Prometheus handler
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: doubleprom
  namespace: istio-system
spec:
  actions:
  - handler: doublehandler.prometheus
    instances:
    - doublerequestcount.metric

上記ファイルを読み込む。

$ kubectl apply -f ./new_telemetry.yaml
metric.config.istio.io/doublerequestcount created
prometheus.config.istio.io/doublehandler created
rule.config.istio.io/doubleprom created

続いて、prometheusのUIにログインする。
下記コマンドを実行して、ブラウザからローカルホスト9090にアクセス。

$ kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090 &

すると、下記のような画面が出て、メトリックがprometheusに送信されていることがわかる。

Isito prometheus

productpageにアクセスすると、エントリーが2ずつ増えていくことがわかる。
このようにどれだけアクセスが来たか？などをistioから取得することが可能。

公式のデモ(ロギング)

下記のロギングを追加。
newlogは新しいログの定義となり、source,user,destination,responseCode,responsesize,latencyの情報を出力する。また、これらのログはseverityがwarningで出力される。
続いて、newhandlerにて先程のログどう処理するか記載し、newlogstdioにてlogentryとstdioのhandlerを結びつける。

---
# Configuration for logentry instances
apiVersion: "config.istio.io/v1alpha2"
kind: logentry
metadata:
  name: newlog
  namespace: istio-system
spec:
  severity: '"warning"'
  timestamp: request.time
  variables:
    source: source.labels["app"] | source.workload.name | "unknown"
    user: source.user | "unknown"
    destination: destination.labels["app"] | destination.workload.name | "unknown"
    responseCode: response.code | 0
    responseSize: response.size | 0
    latency: response.duration | "0ms"
  monitored_resource_type: '"UNSPECIFIED"'
---
# Configuration for a stdio handler
apiVersion: "config.istio.io/v1alpha2"
kind: stdio
metadata:
  name: newhandler
  namespace: istio-system
spec:
 severity_levels:
   warning: 1 # Params.Level.WARNING
 outputAsJson: true
---
# Rule to send logentry instances to a stdio handler
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: newlogstdio
  namespace: istio-system
spec:
  match: "true" # match for all requests
  actions:
   - handler: newhandler.stdio
     instances:
     - newlog.logentry

上記ファイルを読み込む。

$ kubectl apply -f new_logging.yaml
logentry.config.istio.io/newlog created
stdio.config.istio.io/newhandler created
rule.config.istio.io/newlogstdio created

今後デモアプリケーションにアクセスするたびに、ログが追加される。
試しに、下記コマンドでログをフォローして表示してみる。

$ kubectl -n istio-system logs -f --tail=1 $(kubectl -n istio-system get pods -l istio-mixer-type=telemetry -o jsonpath='{.items[0].metadata.name}') -c mixer | grep \"instance\":\"newlog.logentry.istio-system\"

この状態でデモアプリケーションにアクセスすると、下記のようなログが見える。

{"level":"warn","time":"2018-11-14T01:46:26.528539Z","instance":"newlog.logentry.istio-system","destination":"policy","latency":"2.109624ms","responseCode":200,"responseSize":76,"source":"istio-ingressgateway","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:26.532581Z","instance":"newlog.logentry.istio-system","destination":"policy","latency":"1.457536ms","responseCode":200,"responseSize":70,"source":"productpage","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:26.564259Z","instance":"newlog.logentry.istio-system","destination":"policy","latency":"1.813545ms","responseCode":200,"responseSize":70,"source":"reviews","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:26.542126Z","instance":"newlog.logentry.istio-system","destination":"policy","latency":"1.465917ms","responseCode":200,"responseSize":70,"source":"details","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:26.502834Z","instance":"newlog.logentry.istio-system","destination":"details","latency":"9.486802ms","responseCode":200,"responseSize":178,"source":"productpage","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:26.540419Z","instance":"newlog.logentry.istio-system","destination":"details","latency":"11.094098ms","responseCode":200,"responseSize":178,"source":"productpage","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:26.558549Z","instance":"newlog.logentry.istio-system","destination":"reviews","latency":"70.776126ms","responseCode":200,"responseSize":379,"source":"productpage","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:26.613025Z","instance":"newlog.logentry.istio-system","destination":"policy","latency":"1.909275ms","responseCode":200,"responseSize":70,"source":"ratings","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:26.611326Z","instance":"newlog.logentry.istio-system","destination":"ratings","latency":"6.00327ms","responseCode":200,"responseSize":48,"source":"reviews","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:26.609717Z","instance":"newlog.logentry.istio-system","destination":"ratings","latency":"9.143436ms","responseCode":200,"responseSize":48,"source":"reviews","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:26.561753Z","instance":"newlog.logentry.istio-system","destination":"reviews","latency":"67.768853ms","responseCode":200,"responseSize":379,"source":"productpage","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:26.531808Z","instance":"newlog.logentry.istio-system","destination":"productpage","latency":"100.996716ms","responseCode":200,"responseSize":5723,"source":"istio-ingressgateway","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:26.526482Z","instance":"newlog.logentry.istio-system","destination":"productpage","latency":"106.755575ms","responseCode":200,"responseSize":5723,"source":"istio-ingressgateway","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:27.494273Z","instance":"newlog.logentry.istio-system","destination":"telemetry","latency":"4.480481ms","responseCode":200,"responseSize":5,"source":"policy","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:27.505612Z","instance":"newlog.logentry.istio-system","destination":"telemetry","latency":"2.642202ms","responseCode":200,"responseSize":5,"source":"policy","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:27.578369Z","instance":"newlog.logentry.istio-system","destination":"telemetry","latency":"2.865559ms","responseCode":200,"responseSize":5,"source":"policy","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:27.513656Z","instance":"newlog.logentry.istio-system","destination":"telemetry","latency":"2.688707ms","responseCode":200,"responseSize":5,"source":"details","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:27.584923Z","instance":"newlog.logentry.istio-system","destination":"telemetry","latency":"2.569318ms","responseCode":200,"responseSize":5,"source":"reviews","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:27.588581Z","instance":"newlog.logentry.istio-system","destination":"telemetry","latency":"2.368278ms","responseCode":200,"responseSize":5,"source":"reviews","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:27.516006Z","instance":"newlog.logentry.istio-system","destination":"telemetry","latency":"3.341842ms","responseCode":200,"responseSize":5,"source":"productpage","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:27.593452Z","instance":"newlog.logentry.istio-system","destination":"telemetry","latency":"2.107356ms","responseCode":200,"responseSize":5,"source":"productpage","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:27.580069Z","instance":"newlog.logentry.istio-system","destination":"telemetry","latency":"2.786726ms","responseCode":200,"responseSize":5,"source":"ratings","user":"unknown"}
{"level":"warn","time":"2018-11-14T01:46:27.593273Z","instance":"newlog.logentry.istio-system","destination":"telemetry","latency":"3.614374ms","responseCode":200,"responseSize":5,"source":"istio-ingressgateway","user":"unknown"}

これによって、アプリケーションのログを取得することが可能。
障害発生時にどこが悪かったかなどのことが後ほど解析できる。もしくは、ログをベースに監視も可能となる。

おわりに

簡単に監視関連の機能を試してみた。
次はistioをどう使用していくか考えてみる。

コメントを残す

メールアドレスが公開されることはありません。 * が付いている欄は必須項目です

このサイトはスパムを低減するために Akismet を使っています。コメントデータの処理方法の詳細はこちらをご覧ください。

前の記事

IstioのIntelligent Routingを試してみた

次の記事

運用を考慮したKubernetesを構築する