1. 首頁
  2. 問答
  3. kubernetes：無法從其他機器訪問NodePort

kubernetes：無法從其他機器訪問NodePort

2017-10-10 31 views
1

我正在嘗試安裝kubernetes。 檢查在kubernetes上執行的容器時出現問題。 我將服務的類型設置爲Nodeport，但我無法從運行容器的節點以外的節點訪問。 想要讓它可以從其他電腦訪問，請告訴我它在哪裏不同。 我試過externalIPs和LoadBarancer，但這是不可能的。kubernetes：無法從其他機器訪問NodePort

環境

  • 操作系統：Ubuntu的LTS 16.04
  • Kubernetes：1.8
  • 泊塢窗：17.09.0-CE
  • ETCD：3.2.8
  • 法蘭絨：0.9.0

網絡

  • 物理：10.1.1.0/24
  • 絨布：172.16.0.0/16
  • 泊塢窗：192.168.0.0/16

機

  • 主節點（2nodes）： 10.1.1.24,10.1.1.25
  • 工作節點（2nodes）：10.1.1.26,10.1.1.27

kubectl描述SVC nginx的羣集

Name:    nginx-cluster 
Namespace:   default 
Labels:   app=nginx-demo 
Annotations:  <none> 
Selector:   app=nginx-demo 
Type:    ClusterIP 
IP:    172.16.236.159 
Port:    <unset> 8090/TCP 
TargetPort:  80/TCP 
Endpoints:   192.168.24.2:80 
Session Affinity: None 
Events:   <none>

kubectl描述SVC nginx的服務

Name:      nginx-service 
Namespace:    default 
Labels:     app=nginx-demo 
Annotations:    <none> 
Selector:     app=nginx-demo 
Type:      NodePort 
IP:      172.16.199.69 
Port:      <unset> 8090/TCP 
TargetPort:    80/TCP 
NodePort:     <unset> 31659/TCP 
Endpoints:    192.168.24.2:80 
Session Affinity:   None 
External Traffic Policy: Cluster 
Events:     <none>

運行容器工作節點（10.1.1.27）

捲曲10.1.1.27:31659

<!DOCTYPE html> 
<html> 
<head> 
<title>Welcome to nginx!</title> 
...

工人節點（10.1.1.26）

捲曲10.1.1.27:31659

curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.

其它機器（10.1.1.XX）

捲曲10.1.1。27：31659

curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.

kubectl得到莢寬-o

NAME       READY  STATUS RESTARTS AGE  IP    NODE 
echoserver-848b75d85-9fx7r 1/1  Running 3   6d  192.168.70.2 k8swrksv01 
nginx-demo-85cc49574c-wv2b9 1/1  Running 3   6d  192.168.2.2 k8swrksv02

kubectl得到SVC -o寬

NAME   TYPE  CLUSTER-IP  EXTERNAL-IP PORT(S)   AGE  SELECTOR 
clusterip  ClusterIP 172.16.39.77  <none>  80/TCP   6d  run=echoserver 
kubernetes  ClusterIP 172.16.0.1  <none>  443/TCP   10d  <none> 
nginx-cluster ClusterIP 172.16.236.159 <none>  8090/TCP   6d  app=nginx-demo 
nginx-service NodePort 172.16.199.69 <none>  8090:31659/TCP 6d  app=nginx-demo 
nodeport  NodePort 172.16.38.40  <none>  80:31317/TCP  6d  run=echoserver

netstat的-ntlp

tcp  0  0 127.0.0.1:10248   0.0.0.0:*    LISTEN  1963/kubelet 
tcp  0  0 127.0.0.1:10249   0.0.0.0:*    LISTEN  2202/kube-proxy 
tcp  0  0 127.0.0.1:4243   0.0.0.0:*    LISTEN  1758/dockerd 
tcp  0  0 0.0.0.0:22    0.0.0.0:*    LISTEN  996/sshd 
tcp6  0  0 :::4194     :::*     LISTEN  1963/kubelet 
tcp6  0  0 :::10250    :::*     LISTEN  1963/kubelet 
tcp6  0  0 :::31659    :::*     LISTEN  2202/kube-proxy 
tcp6  0  0 :::10255    :::*     LISTEN  1963/kubelet 
tcp6  0  0 :::10256    :::*     LISTEN  2202/kube-proxy 
tcp6  0  0 :::31317    :::*     LISTEN  2202/kube-proxy 
tcp6  0  0 :::22     :::*     LISTEN  996/sshd

iptables-save命令

*nat 
:PREROUTING ACCEPT [0:0] 
:INPUT ACCEPT [0:0] 
:OUTPUT ACCEPT [0:0] 
:POSTROUTING ACCEPT [0:0] 
:DOCKER - [0:0] 
:KUBE-MARK-DROP - [0:0] 
:KUBE-MARK-MASQ - [0:0] 
:KUBE-NODEPORTS - [0:0] 
:KUBE-POSTROUTING - [0:0] 
:KUBE-SEP-AZ4EGFEAU4RTSLJO - [0:0] 
:KUBE-SEP-C7HQKKO26GIFOZZM - [0:0] 
:KUBE-SEP-EWKNS2YCPXGJCXDC - [0:0] 
:KUBE-SEP-LQVPUPFGW6BWATIP - [0:0] 
:KUBE-SEP-OMMOFZ27GPKZ4OPA - [0:0] 
:KUBE-SEP-UD3HOGDD5NDLNY74 - [0:0] 
:KUBE-SERVICES - [0:0] 
:KUBE-SVC-CQNAS6RSUGJF2C2D - [0:0] 
:KUBE-SVC-GKN7Y2BSGW4NJTYL - [0:0] 
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0] 
:KUBE-SVC-XP7QDA4CRQ2QA33W - [0:0] 
:KUBE-SVC-Z5P6OMNAEVLAQUTS - [0:0] 
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES 
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER 
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES 
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER 
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING 
-A POSTROUTING -s 192.168.2.0/24 ! -o docker0 -j MASQUERADE 
-A POSTROUTING -s 192.168.0.0/16 -d 192.168.0.0/16 -j RETURN 
-A POSTROUTING -s 192.168.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE 
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.2.0/24 -j RETURN 
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.0.0/16 -j MASQUERADE 
-A DOCKER -i docker0 -j RETURN 
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000 
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000 
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-MARK-MASQ 
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-SVC-GKN7Y2BSGW4NJTYL 
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-MARK-MASQ 
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-SVC-XP7QDA4CRQ2QA33W 
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE 
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -s 192.168.70.2/32 -m comment --comment "default/clusterip:" -j KUBE-MARK-MASQ 
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -p tcp -m comment --comment "default/clusterip:" -m tcp -j DNAT --to-destination 192.168.70.2:8080 
-A KUBE-SEP-C7HQKKO26GIFOZZM -s 192.168.70.2/32 -m comment --comment "default/nodeport:" -j KUBE-MARK-MASQ 
-A KUBE-SEP-C7HQKKO26GIFOZZM -p tcp -m comment --comment "default/nodeport:" -m tcp -j DNAT --to-destination 192.168.70.2:8080 
-A KUBE-SEP-EWKNS2YCPXGJCXDC -s 10.1.1.25/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ 
-A KUBE-SEP-EWKNS2YCPXGJCXDC -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.25:6443 
-A KUBE-SEP-LQVPUPFGW6BWATIP -s 192.168.2.2/32 -m comment --comment "default/nginx-service:" -j KUBE-MARK-MASQ 
-A KUBE-SEP-LQVPUPFGW6BWATIP -p tcp -m comment --comment "default/nginx-service:" -m tcp -j DNAT --to-destination 192.168.2.2:80 
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -s 10.1.1.24/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ 
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.24:6443 
-A KUBE-SEP-UD3HOGDD5NDLNY74 -s 192.168.2.2/32 -m comment --comment "default/nginx-cluster:" -j KUBE-MARK-MASQ 
-A KUBE-SEP-UD3HOGDD5NDLNY74 -p tcp -m comment --comment "default/nginx-cluster:" -m tcp -j DNAT --to-destination 192.168.2.2:80 
-A KUBE-SERVICES -d 172.16.236.159/32 -p tcp -m comment --comment "default/nginx-cluster: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-Z5P6OMNAEVLAQUTS 
-A KUBE-SERVICES -d 172.16.199.69/32 -p tcp -m comment --comment "default/nginx-service: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-GKN7Y2BSGW4NJTYL 
-A KUBE-SERVICES -d 172.16.38.40/32 -p tcp -m comment --comment "default/nodeport: cluster IP" -m tcp --dport 80 -j KUBE-SVC-XP7QDA4CRQ2QA33W 
-A KUBE-SERVICES -d 172.16.39.77/32 -p tcp -m comment --comment "default/clusterip: cluster IP" -m tcp --dport 80 -j KUBE-SVC-CQNAS6RSUGJF2C2D 
-A KUBE-SERVICES -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y 
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS 
-A KUBE-SVC-CQNAS6RSUGJF2C2D -m comment --comment "default/clusterip:" -j KUBE-SEP-AZ4EGFEAU4RTSLJO 
-A KUBE-SVC-GKN7Y2BSGW4NJTYL -m comment --comment "default/nginx-service:" -j KUBE-SEP-LQVPUPFGW6BWATIP 
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -j KUBE-SEP-OMMOFZ27GPKZ4OPA 
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -j KUBE-SEP-EWKNS2YCPXGJCXDC 
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OMMOFZ27GPKZ4OPA 
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-EWKNS2YCPXGJCXDC 
-A KUBE-SVC-XP7QDA4CRQ2QA33W -m comment --comment "default/nodeport:" -j KUBE-SEP-C7HQKKO26GIFOZZM 
-A KUBE-SVC-Z5P6OMNAEVLAQUTS -m comment --comment "default/nginx-cluster:" -j KUBE-SEP-UD3HOGDD5NDLNY74 
COMMIT 
*filter 
:INPUT ACCEPT [40:14606] 
:FORWARD DROP [0:0] 
:OUTPUT ACCEPT [42:6275] 
:DOCKER - [0:0] 
:DOCKER-ISOLATION - [0:0] 
:DOCKER-USER - [0:0] 
:KUBE-FIREWALL - [0:0] 
:KUBE-SERVICES - [0:0] 
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES 
-A INPUT -j KUBE-FIREWALL 
-A FORWARD -j DOCKER-USER 
-A FORWARD -j DOCKER-ISOLATION 
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 
-A FORWARD -o docker0 -j DOCKER 
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT 
-A FORWARD -i docker0 -o docker0 -j ACCEPT 
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES 
-A OUTPUT -j KUBE-FIREWALL 
-A DOCKER-ISOLATION -j RETURN 
-A DOCKER-USER -j RETURN 
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP 
COMMIT

來源

2017-10-10 Y.Yama

+0

您與哪個提供商合作？ GKE？ – iamnat

+0

哦，我忘了寫。不使用雲。並沒有使用kubeadm，防火牆（ufw）被禁用。 –

回答

1

這是由F的默認DROP造成的ORWARD鏈（反過來是由碼頭造成的）。

如果您將iptables -A FORWARD -j ACCEPT規則添加到您的節點，您可以再次看到它工作。

k8s問題在這裏：https://github.com/kubernetes/kubernetes/issues/39823但實際的修復在這裏https://github.com/kubernetes/kubernetes/pull/52569（預計在1.9）。

來源

2017-12-03 12:56:50 Farcaller

+0

謝謝！ ！ 這是原因。儘管我摧毀了發佈時間的環境，但我試圖在另一個環境中創建它，並且毫無問題地解決它！ –

-1

你可以先檢查節點端口的端口是否打開，

使用

netstat -ntlp

檢查，如果是，那也許有些問題用的iptable或路由，

如果沒有，請檢查防火牆或其他問題

好lu ck

來源

2017-10-14 08:48:02 sam

+0

謝謝！ 添加了netstat和iptables-save的結果。 此端口已打開。 並檢查了iptables的內容。但是，我認爲這個內容是正確的... –

+0

你可以遵循這個規則，首先是'-A KUBE-NODEPORTS -p tcp -m comment --comment「default/nginx-service：」-m tcp --dport 31659 -j KUBE-SVC-GKN7Y2BSGW4NJTYL「表示31659將會轉到'KUBE-SVC-GKN7Y2BSGW4NJTYL'，然後是'-A KUBE-SVC-GKN7Y2BSGW4NJTYL -m註釋 - 註釋「default/nginx-service：」-j KUBE-SEP-LQVPUPFGW6BWATIP'，它告訴你它會轉移到這個地方，最後'-A KUBE-SEP-LQVPUPFGW6BWATIP -p tcp -m註釋 - 註釋「default/nginx-service：」-m tcp -j DNAT - to-destination 192.168.2.2：80'，告訴您的請求將轉到192.168.2.2:80，並檢查該吊艙是否相同？ – sam

+0

並且你可以運行一個ubuntu來檢查集羣ip端口是否工作呢？如果你按照這一步找到解決方案，請給我一個肯定的評論，我不知道爲什麼有人給我負面的評論，說問題的解決方案是總是喜歡這個.....我不是上帝可以直接找到答案.... – sam

0

附加信息。 東西應該阻止端口，但未知...

吊艙正在運行的節點

的nmap 10.1.1.27 -p31000-32000

Not shown: 999 closed ports 
PORT  STATE SERVICE 
31317/tcp open unknown 
31659/tcp open unknown

其他節點

NMAP 10.1 .1.27 -p31000-32000

Not shown: 999 closed ports 
PORT  STATE SERVICE 
31317/tcp filtered unknown 
31659/tcp filtered unknown

來源

2017-10-19 00:06:02

0

Farcaller是正確的。我們在運行firewalld的centos上遇到同樣的情況。

在我們升級到k8s 1.9之前，我們添加了以下firewalld規則。該規則類似於k8中的kube-proxy創建的規則1.9

#!/bin/bash 
# follows https://github.com/kubernetes/kubernetes/pull/52569 introduced in k8s 1.9 
# required to support nodeport services routing from all nodes in the cluster when the firewall is turned on. 
# KUBE-MARK-MASQ corresponds to kube-proxy --iptables-masquerade-bit=14, which is the default. 
KUBE_MARK_MASQ="0x4000/0x4000" 
firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 3 -m mark --mark "$KUBE_MARK_MASQ" -j ACCEPT

來源

2018-02-16 08:59:27

相關問題

 相關問題