{"body":"[rwellum@k8s ~]$ kubectl describe po kube-dns-3913472980-h457k -n kube-system\nName:\t\tkube-dns-3913472980-h457k\nNamespace:\tkube-system\nNode:\t\tk8s/10.240.43.175\nStart Time:\tWed, 17 May 2017 09:16:10 -0400\nLabels:\t\tk8s-app=kube-dns\n\t\tpod-template-hash=3913472980\nAnnotations:\tkubernetes.io/created-by={\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"kube-system\",\"name\":\"kube-dns-3913472980\",\"uid\":\"b4e46ff0-3b02-11e7-9672-52540052...\n\t\tscheduler.alpha.kubernetes.io/critical-pod=\nStatus:\t\tRunning\nIP:\t\t\nControllers:\tReplicaSet/kube-dns-3913472980\nContainers:\n kubedns:\n Container ID:\tdocker://307e39d9213d946e45689c25b6f44c7f8c62001c9bc14b38342773f02f0ca84d\n Image:\t\tgcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1\n Image ID:\t\tdocker-pullable://gcr.io/google_containers/k8s-dns-kube-dns-amd64@sha256:33914315e600dfb756e550828307dfa2b21fb6db24fe3fe495e33d1022f9245d\n Ports:\t\t10053/UDP, 10053/TCP, 10055/TCP\n Args:\n --domain=cluster.local.\n --dns-port=10053\n --config-dir=/kube-dns-config\n --v=2\n State:\t\tWaiting\n Reason:\t\trpc error: code = 2 desc = failed to start container \"307e39d9213d946e45689c25b6f44c7f8c62001c9bc14b38342773f02f0ca84d\": Error response from daemon: {\"message\":\"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827\"}\n Last State:\t\tTerminated\n Reason:\t\tContainerCannotRun\n Message:\t\tcannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827\n Exit Code:\t128\n Started:\t\tMon, 01 Jan 0001 00:00:00 +0000\n Finished:\t\tWed, 17 May 2017 09:19:19 -0400\n Ready:\t\tFalse\n Restart Count:\t0\n Limits:\n memory:\t170Mi\n Requests:\n cpu:\t100m\n memory:\t70Mi\n Liveness:\thttp-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s #success=1 #failure=5\n Readiness:\thttp-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3\n Environment:\n PROMETHEUS_PORT:\t10055\n Mounts:\n /kube-dns-config from kube-dns-config (rw)\n /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-scw9p (ro)\n dnsmasq:\n Container ID:\tdocker://2f7f262c90a351452b3015c8a9901804843798bd98ef0173a8a5ba7623076045\n Image:\t\tgcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1\n Image ID:\t\tdocker-pullable://gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64@sha256:89c9a1d3cfbf370a9c1a949f39f92c1dc2dbe8c3e6cc1802b7f2b48e4dfe9a9e\n Ports:\t\t53/UDP, 53/TCP\n Args:\n -v=2\n -logtostderr\n -configDir=/etc/k8s/dns/dnsmasq-nanny\n -restartDnsmasq=true\n --\n -k\n --cache-size=1000\n --log-facility=-\n --server=/cluster.local/127.0.0.1#10053\n --server=/in-addr.arpa/127.0.0.1#10053\n --server=/ip6.arpa/127.0.0.1#10053\n State:\t\tWaiting\n Reason:\t\trpc error: code = 2 desc = failed to start container \"2f7f262c90a351452b3015c8a9901804843798bd98ef0173a8a5ba7623076045\": Error response from daemon: {\"message\":\"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827\"}\n Last State:\t\tTerminated\n Reason:\t\tContainerCannotRun\n Message:\t\tcannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827\n Exit Code:\t128\n Started:\t\tMon, 01 Jan 0001 00:00:00 +0000\n Finished:\t\tWed, 17 May 2017 09:19:20 -0400\n Ready:\t\tFalse\n Restart Count:\t0\n Requests:\n cpu:\t\t150m\n memory:\t\t20Mi\n Liveness:\t\thttp-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5\n Environment:\t\n Mounts:\n /etc/k8s/dns/dnsmasq-nanny from kube-dns-config (rw)\n /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-scw9p (ro)\n sidecar:\n Container ID:\tdocker://b49abb18c677015610a9b3b4dc70ae1ec18ccd9af7d1ac1357294ba6642bf823\n Image:\t\tgcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1\n Image ID:\t\tdocker-pullable://gcr.io/google_containers/k8s-dns-sidecar-amd64@sha256:d33a91a5d65c223f410891001cd379ac734d036429e033865d700a4176e944b0\n Port:\t\t10054/TCP\n Args:\n --v=2\n --logtostderr\n --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A\n --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A\n State:\t\tWaiting\n Reason:\t\trpc error: code = 2 desc = failed to start container \"b49abb18c677015610a9b3b4dc70ae1ec18ccd9af7d1ac1357294ba6642bf823\": Error response from daemon: {\"message\":\"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827\"}\n Last State:\t\tTerminated\n Reason:\t\tContainerCannotRun\n Message:\t\tcannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827\n Exit Code:\t128\n Started:\t\tMon, 01 Jan 0001 00:00:00 +0000\n Finished:\t\tWed, 17 May 2017 09:19:17 -0400\n Ready:\t\tFalse\n Restart Count:\t0\n Requests:\n cpu:\t\t10m\n memory:\t\t20Mi\n Liveness:\t\thttp-get http://:10054/metrics delay=60s timeout=5s period=10s #success=1 #failure=5\n Environment:\t\n Mounts:\n /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-scw9p (ro)\nConditions:\n Type\t\tStatus\n Initialized \tTrue \n Ready \tFalse \n PodScheduled \tTrue \nVolumes:\n kube-dns-config:\n Type:\tConfigMap (a volume populated by a ConfigMap)\n Name:\tkube-dns\n Optional:\ttrue\n kube-dns-token-scw9p:\n Type:\tSecret (a volume populated by a Secret)\n SecretName:\tkube-dns-token-scw9p\n Optional:\tfalse\nQoS Class:\tBurstable\nNode-Selectors:\t\nTolerations:\tCriticalAddonsOnly=:Exists\n\t\tnode-role.kubernetes.io/master=:NoSchedule\n\t\tnode.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s\n\t\tnode.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s\nEvents:\n FirstSeen\tLastSeen\tCount\tFrom\t\t\tSubObjectPath\tType\t\tReason\t\t\tMessage\n ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\n 5m\t\t4m\t\t8\tdefault-scheduler\t\t\tWarning\t\tFailedScheduling\tno nodes available to schedule pods\n 3m\t\t3m\t\t1\tdefault-scheduler\t\t\tNormal\t\tScheduled\t\tSuccessfully assigned kube-dns-3913472980-h457k to k8s\n 1m\t\t1m\t\t1\tkubelet, k8s\t\t\t\tWarning\t\tFailedSync\t\tError syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: 483938996c0fdc0e7ce894f7e1214f6b0a45960640fbac7b6a9db5c65be5c77b\n 51s\t\t51s\t\t1\tkubelet, k8s\t\t\t\tWarning\t\tFailedSync\t\tError syncing pod, skipping: failed to \"KillPodSandbox\" for \"b4eaec1d-3b02-11e7-9672-52540052d3ab\" with KillPodSandboxError: \"rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \\\"kube-dns-3913472980-h457k_kube-system\\\" network: CNI failed to retrieve network namespace path: Error response from daemon: {\\\"message\\\":\\\"devmapper: Unknown device bca34cfee54b429563f75529cc2eadc985e3e5d5f25ca10f15eb5bf19e210c6f\\\"}\"\n\n 49s\t49s\t1\tkubelet, k8s\t\tWarning\tFailedSync\tError syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: 8af6d91aaec099300a04d0fb1f9e042bcaa484d2596952696e1df5116c28565f\n 3m\t21s\t31\tkubelet, k8s\t\tWarning\tFailedSync\tError syncing pod, skipping: failed to \"CreatePodSandbox\" for \"kube-dns-3913472980-h457k_kube-system(b4eaec1d-3b02-11e7-9672-52540052d3ab)\" with CreatePodSandboxError: \"CreatePodSandbox for pod \\\"kube-dns-3913472980-h457k_kube-system(b4eaec1d-3b02-11e7-9672-52540052d3ab)\\\" failed: rpc error: code = 2 desc = NetworkPlugin cni failed to set up pod \\\"kube-dns-3913472980-h457k_kube-system\\\" network: no IP addresses available in network: k8s-pod-network\"\n\n 20s\t20s\t1\tkubelet, k8s\tspec.containers{sidecar}\tNormal\tPulled\t\tContainer image \"gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1\" already present on machine\n 19s\t19s\t1\tkubelet, k8s\tspec.containers{sidecar}\tNormal\tCreated\t\tCreated container with id b49abb18c677015610a9b3b4dc70ae1ec18ccd9af7d1ac1357294ba6642bf823\n 18s\t18s\t1\tkubelet, k8s\tspec.containers{sidecar}\tWarning\tFailed\t\tFailed to start container with id b49abb18c677015610a9b3b4dc70ae1ec18ccd9af7d1ac1357294ba6642bf823 with error: rpc error: code = 2 desc = failed to start container \"b49abb18c677015610a9b3b4dc70ae1ec18ccd9af7d1ac1357294ba6642bf823\": Error response from daemon: {\"message\":\"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827\"}\n 18s\t18s\t1\tkubelet, k8s\tspec.containers{kubedns}\tNormal\tPulled\t\tContainer image \"gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1\" already present on machine\n 17s\t17s\t1\tkubelet, k8s\tspec.containers{kubedns}\tNormal\tCreated\t\tCreated container with id 307e39d9213d946e45689c25b6f44c7f8c62001c9bc14b38342773f02f0ca84d\n 17s\t17s\t1\tkubelet, k8s\tspec.containers{kubedns}\tWarning\tFailed\t\tFailed to start container with id 307e39d9213d946e45689c25b6f44c7f8c62001c9bc14b38342773f02f0ca84d with error: rpc error: code = 2 desc = failed to start container \"307e39d9213d946e45689c25b6f44c7f8c62001c9bc14b38342773f02f0ca84d\": Error response from daemon: {\"message\":\"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827\"}\n 17s\t17s\t1\tkubelet, k8s\tspec.containers{dnsmasq}\tNormal\tPulled\t\tContainer image \"gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1\" already present on machine\n 15s\t15s\t1\tkubelet, k8s\tspec.containers{dnsmasq}\tNormal\tCreated\t\tCreated container with id 2f7f262c90a351452b3015c8a9901804843798bd98ef0173a8a5ba7623076045\n 15s\t15s\t1\tkubelet, k8s\tspec.containers{dnsmasq}\tWarning\tFailed\t\tFailed to start container with id 2f7f262c90a351452b3015c8a9901804843798bd98ef0173a8a5ba7623076045 with error: rpc error: code = 2 desc = failed to start container \"2f7f262c90a351452b3015c8a9901804843798bd98ef0173a8a5ba7623076045\": Error response from daemon: {\"message\":\"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827\"}\n 15s\t15s\t1\tkubelet, k8s\t\t\t\t\tWarning\tFailedSync\tError syncing pod, skipping: [failed to \"StartContainer\" for \"sidecar\" with rpc error: code = 2 desc = failed to start container \"b49abb18c677015610a9b3b4dc70ae1ec18ccd9af7d1ac1357294ba6642bf823\": Error response from daemon: {\"message\":\"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827\"}: \"Start Container Failed\"\n, failed to \"StartContainer\" for \"kubedns\" with rpc error: code = 2 desc = failed to start container \"307e39d9213d946e45689c25b6f44c7f8c62001c9bc14b38342773f02f0ca84d\": Error response from daemon: {\"message\":\"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827\"}: \"Start Container Failed\"\n, failed to \"StartContainer\" for \"dnsmasq\" with rpc error: code = 2 desc = failed to start container \"2f7f262c90a351452b3015c8a9901804843798bd98ef0173a8a5ba7623076045\": Error response from daemon: {\"message\":\"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827\"}: \"Start Container Failed\"\n]\n 3m\t2s\t32\tkubelet, k8s\t\tNormal\tSandboxChanged\tPod sandbox changed, it will be killed and re-created.\n[rwellum@k8s ~]$ \n","name":"","extension":"txt","url":"https://www.irccloud.com/pastebin/cJXd0NFH","modified":1495027254,"id":"cJXd0NFH","size":11254,"lines":153,"own_paste":false,"theme":"","date":1495027254}