普羅米修斯 - Kubernetes RBAC

Question

我我GKE API服務器升級到1.6，我在升級節點到1.6的過程中，卻遇到了一個問題...

我有一個普羅米修斯服務器（1.5版本。 2）運行在由Kubernetes部署管理的Pod上，其中有一些節點運行版本1.5.4 Kubelet，其中一個新節點運行1.6。

Prometheus無法連接到新節點 - 指標端點返回401未經授權。

這似乎是一個RBAC問題，但我不知道如何繼續。我找不到有關Prometheus服務器需要什麼角色的文檔，甚至不知道如何將它們授予服務器。

從coreos /普羅米修斯運營商回購我能拼湊出我所料工作的配置：

apiVersion: v1 
kind: ServiceAccount 
metadata: 
    name: prometheus 
--- 

apiVersion: rbac.authorization.k8s.io/v1beta1 
kind: ClusterRole 
metadata: 
    name: prometheus 
rules: 
- apiGroups: [""] 
    resources: 
    - nodes 
    - services 
    - endpoints 
    - pods 
    verbs: ["get", "list", "watch"] 
- apiGroups: [""] 
    resources: 
    - configmaps 
    verbs: ["get"] 
- nonResourceURLs: ["/metrics"] 
    verbs: ["get"] 
--- 

apiVersion: rbac.authorization.k8s.io/v1beta1 
kind: ClusterRoleBinding 
metadata: 
    name: prometheus 
roleRef: 
    apiGroup: rbac.authorization.k8s.io 
    kind: ClusterRole 
    name: prometheus 
subjects: 
- kind: ServiceAccount 
    name: prometheus 
    namespace: default 
--- 

apiVersion: v1 
kind: ServiceAccount 
metadata: 
    name: prometheus 
    namespace: default 
secrets: 
- name: prometheus-token-xxxxx 

--- 
apiVersion: extensions/v1beta1 
kind: Deployment 
metadata: 
    labels: 
    app: prometheus-prometheus 
    component: server 
    release: prometheus 
    name: prometheus-server 
    namespace: default 
spec: 
    replicas: 1 
    selector: 
    matchLabels: 
     app: prometheus-prometheus 
     component: server 
     release: prometheus 
    strategy: 
    rollingUpdate: 
     maxSurge: 1 
     maxUnavailable: 1 
    type: RollingUpdate 
    template: 
    metadata: 
     labels: 
     app: prometheus-prometheus 
     component: server 
     release: prometheus 
    spec: 
     dnsPolicy: ClusterFirst 
     restartPolicy: Always 
     schedulerName: default-scheduler 
     serviceAccount: prometheus 
     serviceAccountName: prometheus 
     ... 

但是普羅米修斯還是日漸401S。

更新：看起來像喬丹所說的kubernetes認證問題。在此查看新的，更具針對性的問題; https://serverfault.com/questions/843751/kubernetes-node-metrics-endpoint-returns-401

Answer 1

根據對@ JorritSalverda的票證的討論; https://github.com/prometheus/prometheus/issues/2606#issuecomment-294869099

由於GKE不允許您獲得允許您使用kubelet進行身份驗證的客戶端證書，因此GKE用戶的最佳解決方案似乎使用kubernetes API服務器作爲代理請求節點。

要做到這一點（引用@JorritSalverda）;

「對於內部GKE運行我的普羅米修斯的服務器我現在已經將其與以下重新貼標籤運行：

relabel_configs: 
- action: labelmap 
    regex: __meta_kubernetes_node_label_(.+) 
- target_label: __address__ 
    replacement: kubernetes.default.svc.cluster.local:443 
- target_label: __scheme__ 
    replacement: https 
- source_labels: [__meta_kubernetes_node_name] 
    regex: (.+) 
    target_label: __metrics_path__ 
    replacement: /api/v1/nodes/${1}/proxy/metrics 

並綁定到普羅米修斯使用的服務帳戶以下ClusterRole：

apiVersion: rbac.authorization.k8s.io/v1beta1 
kind: ClusterRole 
metadata: 
    name: prometheus 
rules: 
- apiGroups: [""] 
    resources: 
    - nodes 
    - nodes/proxy 
    - services 
    - endpoints 
    - pods 
    verbs: ["get", "list", "watch"] 

因爲在RBAC失敗的情況下，GKE集羣仍然具有ABAC後備箱我並不是100％確定，但它涵蓋了所有必需的權限。

Answer 2

401意味着未經認證，這意味着它不是一個RBAC問題。我相信GKE不再允許匿名訪問1.6版的kubelet。你用什麼憑證來認證kubelet？

Answer 3

這是我h ave正在爲角色定義和綁定工作。

apiVersion: rbac.authorization.k8s.io/v1beta1 
 
kind: ClusterRole 
 
metadata: 
 
    name: prometheus 
 
rules: 
 
- apiGroups: [""] 
 
    resources: 
 
    - nodes 
 
    - services 
 
    - endpoints 
 
    - pods 
 
    verbs: ["get", "list", "watch"] 
 
- nonResourceURLs: ["/metrics"] 
 
    verbs: ["get"] 
 
--- 
 
apiVersion: v1 
 
kind: ServiceAccount 
 
metadata: 
 
    name: prometheus 
 
    namespace: default 
 
--- 
 
apiVersion: rbac.authorization.k8s.io/v1beta1 
 
kind: ClusterRoleBinding 
 
metadata: 
 
    name: prometheus 
 
roleRef: 
 
    apiGroup: rbac.authorization.k8s.io 
 
    kind: ClusterRole 
 
    name: prometheus 
 
subjects: 
 
- kind: ServiceAccount 
 
    name: prometheus 
 
    namespace: default