無法在etcd3中找到Kubernetes apiserver的數據

Question

我使用kargo的有用腳本來構建我的羣集。儘管查看了apiserver的詳細日誌，但我無法找到數據存儲在etcd3中的位置。

這是我看到的hyperkube API服務器日誌：

$ docker logs k8s_kube-apiserver.fd19548d_kube-apiserver-kube-master-01_kube-system_2f6ad6b0bf81ca6a0e2b4d499a25fc89_aa25196e 
[[ SNIP ]] 
I0127 23:31:55.871267  1 storage_factory.go:242] storing { podtemplates} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>} 
I0127 23:31:55.875975  1 storage_factory.go:242] storing { events} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>} 
I0127 23:31:55.876169  1 reflector.go:234] Listing and watching *api.PodTemplate from k8s.io/kubernetes/pkg/storage/cacher.go:215 
I0127 23:31:55.877950  1 compact.go:55] compactor already exists for endpoints [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] 
I0127 23:31:55.878148  1 storage_factory.go:242] storing { limitranges} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>} 
I0127 23:31:55.879372  1 compact.go:55] compactor already exists for endpoints [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] 

的hyperkube apiserver開始使用這些參數：

$ docker inspect k8s_kube-apiserver.b6395694_kube-apiserver-kube-master-01_kube-system_2f6ad6b0bf81ca6a0e2b4d499a25fc89_4338b366 
[ 
    { 
     "Id": "33c76fa64bbd5d5a656e329cf87ed3707077659c69dc281127751f594460242b", 
     "Created": "2017-01-27T23:35:10.691147667Z", 
     "Path": "/hyperkube", 
     "Args": [ 
      "apiserver", 
      "--advertise-address=10.60.68.23", 
      "--etcd-servers=https://10.60.68.11:2379,https://10.60.68.39:2379,https://10.60.68.35:2379", 
      "--etcd-quorum-read=true", 
      "--etcd-cafile=/etc/ssl/etcd/ssl/ca.pem", 
      "--etcd-certfile=/etc/ssl/etcd/ssl/node-kube-master-01.pem", 
      "--etcd-keyfile=/etc/ssl/etcd/ssl/node-kube-master-01-key.pem", 
      "--insecure-bind-address=127.0.0.1", 
      "--apiserver-count=3", 
      "--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota", 
      "--service-cluster-ip-range=10.233.0.0/18", 
      "--service-node-port-range=30000-32767", 
      "--client-ca-file=/etc/kubernetes/ssl/ca.pem", 
      "--basic-auth-file=/etc/kubernetes/users/known_users.csv", 
      "--tls-cert-file=/etc/kubernetes/ssl/apiserver.pem", 
      "--tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem", 
      "--token-auth-file=/etc/kubernetes/tokens/known_tokens.csv", 
      "--service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem", 
      "--secure-port=443", 
      "--insecure-port=8080", 
      "--v=4", 
      "--allow-privileged=true", 
      "--cloud-provider=openstack", 
      "--cloud-config=/etc/kubernetes/cloud_config", 
      "--anonymous-auth=False" 
     ], 

無論在哪裏它覆蓋默認ETCD的/registry前綴。我不知道apiserver在哪裏存儲數據。

$ docker exec -it etcd3 etcdctl --peers https://10.60.68.11:2379 ls /registry 
Error: 100: Key not found (/registry) [163] 

Answer 1

如果您的系統設置正確，請注意有一些從etcd2到etcd3的更改。例如，etcd3中的關鍵空間現在是平坦的，與etcd2不同，不再有任何目錄。如上所示，不要使用etcdctl'ls'命令，而應使用'get'命令（並確保etcdctl命令通過設置ETCDCTL_API = 3環境變量來使用v3 api）。

Answer 2

通過kubernetes爲了獲取存儲在ETCD V3鍵和值：

ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get --prefix /registry 

要想從ETCD V3 1個指定鍵和值，例如：基於

ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get /registry/services/specs/default/kubernetes 

：https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/interacting_v3.md

Answer 3

ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get/--prefix --keys-only