2017-02-26 52 views
1

我有一個Kubernetes 1.5.3集羣,它使用kubeadm(詳情如下)在兩個節點上啓動。我一直在試圖啓動今晚的prometheus-operator。在我刪除了一種類型=普羅米修斯的資源之後,我的集羣進入了一個非常奇怪的狀態,當我試圖重新創建普羅米修斯資源時,雖然TPR存在,但沒有創建任何Pod。調查使我:kube-apiserver無法正常運行,「名稱命名空間不存在索引」

[email protected]:~$ kubectl logs -n kube-system kube-apiserver-hypervisor01 | tail -10 W0226 03:57:34.641244 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:57:34.645073 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:04.642150 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:04.647953 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:34.642118 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:34.646427 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:04.642189 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:04.647978 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:34.646129 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:34.666355 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist

解決辦法在此rancher github issue是簡單地重新啓動API服務器,這將是很好,除了這並沒有解決問題。否則,它會指向etcd上的高負載,正如您在下面看到的那樣,該負載是託管爲一個容器,位於當前負載小於1的主機上。etcdctl命令運行成功,導致我相信etcd本身很好。爲了好的措施,我碰到了etcd docker容器,它仍然正常工作。但是,從這裏開始,我迷路了,需要一些幫助讓我的集羣恢復正常工作狀態。

羣集概述:

[email protected]:~$ kubectl get no NAME STATUS AGE hypervisor01 Ready,master 2d kubes01.pss9.kickass.systems Ready 2d [email protected]:~$ kubectl get -n kube-system po NAME READY STATUS RESTARTS AGE dummy-2088944543-2p3zf 1/1 Running 1 2d etcd-hypervisor01 1/1 Running 2 2d kube-apiserver-hypervisor01 1/1 Running 3 2d kube-controller-manager-hypervisor01 1/1 Running 1 2d kube-discovery-1769846148-v8h50 1/1 Running 1 2d kube-dns-2924299975-3s26d 4/4 Running 4 2d kube-proxy-vpw73 1/1 Running 1 2d kube-proxy-zfh13 1/1 Running 0 2d kube-registry-proxy-6hhk9 1/1 Running 1 2d kube-registry-proxy-nl1s1 1/1 Running 0 2d kube-registry-v0-4d94t 1/1 Running 0 2d kube-scheduler-hypervisor01 1/1 Running 1 2d [email protected]:~$ kubectl get po NAME READY STATUS RESTARTS AGE kube-flannel-ds-589fw 2/2 Running 3 2d kube-flannel-ds-7f5sx 2/2 Running 0 2d kube-state-metrics-3229993571-20nbc 1/1 Running 0 1h node-exporter-1cdlj 1/1 Running 0 1h node-exporter-jc54s 1/1 Running 0 1h prometheus-operator-996254120-0wzg6 1/1 Running 0 2h

+0

https://github.com/kubernetes/kubernetes/blob/v1.5.3/pkg/client/cache/listers.go#L68該代碼似乎暗示事情應該繼續在這裏成功,所以也許這個日誌是一個紅鯡魚... –

回答

1

E0226 16:20:47.861762 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-k8s, requeuing: Failed to create prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value I0226 16:20:47.862793 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-k8s", UID:"272d42fd-fbd1-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304212", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0, error: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value E0226 16:20:47.865352 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-services, requeuing: Failed to create prometheus-prometheus-services-db-prometheus-prometheus-services-0: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value I0226 16:20:47.865472 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-services", UID:"1899b833-fbd2-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304733", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-services-db-prometheus-prometheus-services-0, error: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value E0226 16:21:17.854692 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-k8s, requeuing: Failed to create prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value I0226 16:21:17.855043 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-k8s", UID:"272d42fd-fbd1-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304212", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0, error: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value E0226 16:21:17.858436 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-services, requeuing: Failed to create prometheus-prometheus-services-db-prometheus-prometheus-services-0: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value I0226 16:21:17.858561 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-services", UID:"1899b833-fbd2-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304733", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-services-db-prometheus-prometheus-services-0, error: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value

用戶錯誤捂臉,沒想到在管理日誌看。

相關問題