Kubernetes watch APIを試す

Kubernetes wach APIを試してみたので

環境

今回の環境は以下の通り。

$ kubectl get node -o wide
NAME          STATUS    ROLES     AGE       VERSION   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
test-master   Ready     master    3d        v1.11.1   <none>        Ubuntu 18.04.1 LTS   4.15.0-20-generic   docker://17.12.1-ce
test-node1    Ready     <none>    3d        v1.11.1   <none>        Ubuntu 18.04.1 LTS   4.15.0-20-generic   docker://17.12.1-ce
test-node2    Ready     <none>    3d        v1.11.1   <none>        Ubuntu 18.04.1 LTS   4.15.0-20-generic   docker://17.12.1-ce

REST API用サービスアカウントの作成

Namespaceとserviceaccountを作成する

$ kubectl create ns rest-test
namespace "rest-test" created

$  kubectl -n rest-test create serviceaccount rest-client
serviceaccount/rest-client created

kube configを作ってkubectlからアクセスしてみる

現在使用しているKUBECONIFGに新しいcontextを追加して、先程作成したserviceaccountの情報をセットする。
まず新しいcontextの作成。

$ kubectl config set-context rest-client
Context "rest-client" created.

現在のcontextからクラスタの名前を取得する

$ c_context=$(kubectl config current-context)
$ cluster=$(kubectl config view -o jsonpath="{$.contexts[?(@.name == \"$c_context\")].context.cluster}")

その後、serviceaccountのtokenを取得。

$ token_name=$(kubectl -n rest-test get serviceaccount rest-client -o 'jsonpath={.secrets[0].name}')
$ namespace=$(kubectl -n rest-test get secret $token_name -o 'jsonpath={.data.namespace}' | base64 -d)
$ token=$(kubectl -n rest-test get secret $token_name -o 'jsonpath={.data.token}' | base64 -d)

新しいuserを作成し、取得した値をcontextに設定する。

$ kubectl config set-credentials rest-client --token="$token"
User "rest-client" set.
$ kubectl config set-context rest-client --cluster="$cluster" --namespace="$namespace" --user="rest-client"
Context "rest-client" modified.

これでrest-clientのサービスアカウントを使って、情報をアクセスする準備が整った。
試しに、Podの情報を取得してみる。

$ kubectl config use-context rest-client
Switched to context "rest-client".
$ kubectl get pod
No resources found.
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:rest-test:rest-client" cannot list pods in the namespace "rest-test"

何もRoleを設定していないので、当然ながら何も取得できない。

curlを使用してアクセス

前節で取得したtokenを使用

$ curl -k -H "Authorization: Bearer $token" https://test-master:6443/api/v1/namespaces/rest-test/pods
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:rest-test:rest-client\" cannot list pods in the namespace \"rest-test\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}

同様に取得に失敗する。

ここで、以前のkubernetes-admin@kubernetesにcontextを変更してserviceaccountにcluster-admin roleを設定する

$ kubectl -n rest-test create rolebinding rest-api-rolebinding --clusterrole=cluster-admin --serviceaccount=rest-test:rest-client
rolebinding.rbac.authorization.k8s.io/rest-api-rolebinding created

先ほどと同じREST APIを叩くと、取得できる

$ curl -k -H "Authorization: Bearer $token" https://test-master:6443/api/v1/namespaces/rest-test/pods
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/namespaces/rest-test/pods",
    "resourceVersion": "1193942"
  },
  "items": []
}

Watch APIを使用する

試しにwatch APIを使用

$ curl -k -H "Authorization: Bearer $token" https://test-master:6443/api/v1/namespaces/rest-test/pods?watch=true

既に起動済みのPodがいる場合には以下のようなログがでる

{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"test1-66cb5d55d8-ps4j6","generateName":"test1-66cb5d55d8-","namespace":"rest-test","selfLink":"/api/v1/namespaces/rest-test/pods/test1-66cb5d55d8-ps4j6","uid":"ee7520e8-97d5-11e8-a64c-005056a831ff","resourceVersion":"1206714","creationTimestamp":"2018-08-04T11:02:51Z","labels":{"pod-template-hash":"2276181184","run":"test1"},"annotations":{"cni.projectcalico.org/podIP":"10.244.2.5/32"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"test1-66cb5d55d8","uid":"ee6949e7-97d5-11e8-a64c-005056a831ff","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-kgn24","secret":{"secretName":"default-token-kgn24","defaultMode":420}}],"containers":[{"name":"test1","image":"nginx","resources":{},"volumeMounts":[{"name":"default-token-kgn24","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"test-node1","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:02:52Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:03:19Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":null},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:02:51Z"}],"hostIP":"10.16.181.92","podIP":"10.244.2.5","startTime":"2018-08-04T11:02:52Z","containerStatuses":[{"name":"test1","state":{"running":{"startedAt":"2018-08-04T11:03:19Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"nginx:latest","imageID":"docker-pullable://nginx@sha256:d85914d547a6c92faa39ce7058bd7529baacab7e0cd4255442b04577c4d1f424","containerID":"docker://15e65c77afb6d1a453eef4e1d2b2a217e3db1968720d71846b49cf5a634fe41a"}],"qosClass":"BestEffort"}}}

別のwindowで追加のpodを作成するとcurlを起動したwindowに追加のアップデートが以下のように流れる

{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"test3-78f84b5fb4-42x4j","generateName":"test3-78f84b5fb4-","namespace":"rest-test","selfLink":"/api/v1/namespaces/rest-test/pods/test3-78f84b5fb4-42x4j","uid":"28679194-97d7-11e8-a64c-005056a831ff","resourceVersion":"1207444","creationTimestamp":"2018-08-04T11:11:38Z","labels":{"pod-template-hash":"3494061960","run":"test3"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"test3-78f84b5fb4","uid":"2858892b-97d7-11e8-a64c-005056a831ff","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-kgn24","secret":{"secretName":"default-token-kgn24","defaultMode":420}}],"containers":[{"name":"test3","image":"nginx","resources":{},"volumeMounts":[{"name":"default-token-kgn24","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0},"status":{"phase":"Pending","qosClass":"BestEffort"}}}
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"test3-78f84b5fb4-42x4j","generateName":"test3-78f84b5fb4-","namespace":"rest-test","selfLink":"/api/v1/namespaces/rest-test/pods/test3-78f84b5fb4-42x4j","uid":"28679194-97d7-11e8-a64c-005056a831ff","resourceVersion":"1207447","creationTimestamp":"2018-08-04T11:11:38Z","labels":{"pod-template-hash":"3494061960","run":"test3"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"test3-78f84b5fb4","uid":"2858892b-97d7-11e8-a64c-005056a831ff","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-kgn24","secret":{"secretName":"default-token-kgn24","defaultMode":420}}],"containers":[{"name":"test3","image":"nginx","resources":{},"volumeMounts":[{"name":"default-token-kgn24","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"test-node1","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0},"status":{"phase":"Pending","conditions":[{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z"}],"qosClass":"BestEffort"}}}
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"test3-78f84b5fb4-42x4j","generateName":"test3-78f84b5fb4-","namespace":"rest-test","selfLink":"/api/v1/namespaces/rest-test/pods/test3-78f84b5fb4-42x4j","uid":"28679194-97d7-11e8-a64c-005056a831ff","resourceVersion":"1207450","creationTimestamp":"2018-08-04T11:11:38Z","labels":{"pod-template-hash":"3494061960","run":"test3"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"test3-78f84b5fb4","uid":"2858892b-97d7-11e8-a64c-005056a831ff","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-kgn24","secret":{"secretName":"default-token-kgn24","defaultMode":420}}],"containers":[{"name":"test3","image":"nginx","resources":{},"volumeMounts":[{"name":"default-token-kgn24","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"test-node1","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z","reason":"ContainersNotReady","message":"containers with unready status: [test3]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":null,"reason":"ContainersNotReady","message":"containers with unready status: [test3]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z"}],"hostIP":"10.16.181.92","startTime":"2018-08-04T11:11:38Z","containerStatuses":[{"name":"test3","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"nginx","imageID":""}],"qosClass":"BestEffort"}}}
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"test3-78f84b5fb4-42x4j","generateName":"test3-78f84b5fb4-","namespace":"rest-test","selfLink":"/api/v1/namespaces/rest-test/pods/test3-78f84b5fb4-42x4j","uid":"28679194-97d7-11e8-a64c-005056a831ff","resourceVersion":"1207455","creationTimestamp":"2018-08-04T11:11:38Z","labels":{"pod-template-hash":"3494061960","run":"test3"},"annotations":{"cni.projectcalico.org/podIP":"10.244.2.6/32"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"test3-78f84b5fb4","uid":"2858892b-97d7-11e8-a64c-005056a831ff","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-kgn24","secret":{"secretName":"default-token-kgn24","defaultMode":420}}],"containers":[{"name":"test3","image":"nginx","resources":{},"volumeMounts":[{"name":"default-token-kgn24","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"test-node1","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z","reason":"ContainersNotReady","message":"containers with unready status: [test3]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":null,"reason":"ContainersNotReady","message":"containers with unready status: [test3]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z"}],"hostIP":"10.16.181.92","startTime":"2018-08-04T11:11:38Z","containerStatuses":[{"name":"test3","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"nginx","imageID":""}],"qosClass":"BestEffort"}}}
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"test3-78f84b5fb4-42x4j","generateName":"test3-78f84b5fb4-","namespace":"rest-test","selfLink":"/api/v1/namespaces/rest-test/pods/test3-78f84b5fb4-42x4j","uid":"28679194-97d7-11e8-a64c-005056a831ff","resourceVersion":"1207467","creationTimestamp":"2018-08-04T11:11:38Z","labels":{"pod-template-hash":"3494061960","run":"test3"},"annotations":{"cni.projectcalico.org/podIP":"10.244.2.6/32"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"test3-78f84b5fb4","uid":"2858892b-97d7-11e8-a64c-005056a831ff","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-kgn24","secret":{"secretName":"default-token-kgn24","defaultMode":420}}],"containers":[{"name":"test3","image":"nginx","resources":{},"volumeMounts":[{"name":"default-token-kgn24","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"test-node1","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:46Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":null},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z"}],"hostIP":"10.16.181.92","podIP":"10.244.2.6","startTime":"2018-08-04T11:11:38Z","containerStatuses":[{"name":"test3","state":{"running":{"startedAt":"2018-08-04T11:11:45Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"nginx:latest","imageID":"docker-pullable://nginx@sha256:d85914d547a6c92faa39ce7058bd7529baacab7e0cd4255442b04577c4d1f424","containerID":"docker://f26ce653156dc61cc53f0510d49975d5e33ccae200a9b514d345e8fec3cc6ee5"}],"qosClass":"BestEffort"}}}

resourceVersionを指定してwach APIを叩くとそのバージョンより新しいアップデートのみ取得可能。

$ curl -k -H "Authorization: Bearer $token" 'https://test-master:6443/api/v1/namespaces/rest-test/pods?watch=true&resourceVersion=1207450'
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"test3-78f84b5fb4-42x4j","generateName":"test3-78f84b5fb4-","namespace":"rest-test","selfLink":"/api/v1/namespaces/rest-test/pods/test3-78f84b5fb4-42x4j","uid":"28679194-97d7-11e8-a64c-005056a831ff","resourceVersion":"1207455","creationTimestamp":"2018-08-04T11:11:38Z","labels":{"pod-template-hash":"3494061960","run":"test3"},"annotations":{"cni.projectcalico.org/podIP":"10.244.2.6/32"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"test3-78f84b5fb4","uid":"2858892b-97d7-11e8-a64c-005056a831ff","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-kgn24","secret":{"secretName":"default-token-kgn24","defaultMode":420}}],"containers":[{"name":"test3","image":"nginx","resources":{},"volumeMounts":[{"name":"default-token-kgn24","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"test-node1","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z","reason":"ContainersNotReady","message":"containers with unready status: [test3]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":null,"reason":"ContainersNotReady","message":"containers with unready status: [test3]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z"}],"hostIP":"10.16.181.92","startTime":"2018-08-04T11:11:38Z","containerStatuses":[{"name":"test3","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"nginx","imageID":""}],"qosClass":"BestEffort"}}}
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"test3-78f84b5fb4-42x4j","generateName":"test3-78f84b5fb4-","namespace":"rest-test","selfLink":"/api/v1/namespaces/rest-test/pods/test3-78f84b5fb4-42x4j","uid":"28679194-97d7-11e8-a64c-005056a831ff","resourceVersion":"1207467","creationTimestamp":"2018-08-04T11:11:38Z","labels":{"pod-template-hash":"3494061960","run":"test3"},"annotations":{"cni.projectcalico.org/podIP":"10.244.2.6/32"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"test3-78f84b5fb4","uid":"2858892b-97d7-11e8-a64c-005056a831ff","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-kgn24","secret":{"secretName":"default-token-kgn24","defaultMode":420}}],"containers":[{"name":"test3","image":"nginx","resources":{},"volumeMounts":[{"name":"default-token-kgn24","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"test-node1","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:46Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":null},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-04T11:11:38Z"}],"hostIP":"10.16.181.92","podIP":"10.244.2.6","startTime":"2018-08-04T11:11:38Z","containerStatuses":[{"name":"test3","state":{"running":{"startedAt":"2018-08-04T11:11:45Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"nginx:latest","imageID":"docker-pullable://nginx@sha256:d85914d547a6c92faa39ce7058bd7529baacab7e0cd4255442b04577c4d1f424","containerID":"docker://f26ce653156dc61cc53f0510d49975d5e33ccae200a9b514d345e8fec3cc6ee5"}],"qosClass":"BestEffort"}}}