在mesos上運行kafka發佈消息的問題DCOS

Question

我有一個運行DCOS的小型集羣。我可以在this指南中成功安裝kafka。運行

$ dcos kafka connection 

給

{ 
    "address": [ 
    "10.131.17.126:9475", 
    "10.131.24.6:9655", 
    "10.131.14.192:9181" 
    ], 
    "zookeeper": "master.mesos:2181/dcos-service-kafka", 
    "dns": [ 
    "broker-0.kafka.mesos:9475", 
    "broker-1.kafka.mesos:9655", 
    "broker-2.kafka.mesos:9181" 
    ] 
} 

我可以創建主題，我已經研究飼養員用CLI工具和狀態似乎不錯

get /dcos-service-kafka/brokers/ids/0 
{"jmx_port":-1,"timestamp":"1474206074029","endpoints":["PLAINTEXT://10.131.17.126:9475"],"host":"10.131.17.126","version":3,"port":9475} 
get /dcos-service-kafka/brokers/ids/1 
{"jmx_port":-1,"timestamp":"1474206120002","endpoints":["PLAINTEXT://10.131.24.6:9655"],"host":"10.131.24.6","version":3,"port":9655} 
get /dcos-service-kafka/brokers/ids/2 
{"jmx_port":-1,"timestamp":"1474206122985","endpoints":["PLAINTEXT://10.131.14.192:9181"],"host":"10.131.14.192","version":3,"port":9181} 

然而，當我嘗試發佈

echo "Hello, World." | ./kafka-console-producer.sh --broker-list 10.131.17.126:9475, 10.131.24.6:9655, 10.131.14.192:9181 --topic topic1 

我得到

[2016-09-18 18:49:32,909] ERROR Error when sending message to topic topic1 with key: null, value: 13 bytes with error: Failed to update metadata after 60000 ms. (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) 

我懷疑這可能是與私人與公共IP地址，也許host.name在server.properties。

任何人都可以提出一些建議，我可能會調試（並希望修復！）的問題，所以我可以成功地發佈消息？

感謝

AJ

Answer 1

編輯：對於任何人都希望在未來。 This was a problem in /etc/hosts caused by a terraform script。

您的懷疑是正確的。這些是私人IP地址，不能從羣集外尋址。爲了與Kafka進行通信，您需要設置一個VPN，以使這些IP地址變得可訪問，或者在集羣中的某臺計算機上運行發佈命令。

另外，它看起來像是運行在1.8以前的DC/OS版本上。如果使用1.8，則無論動態分配的IP地址如何，您都可以使用更簡單的代理端點。您可以使用命名的VIP broker.kafka.l4lb.thisdcos.directory:9092，但是這隻能從羣集中的機器尋址。

將haproxy或nginx設置爲指向命名VIP也是一種輕鬆外部訪問在DC/OS集羣上運行的服務（本例中爲Kafka）的方法。您需要確保這些代理在公共代理上運行。 See here for more details。

這裏是安裝，生產和從默認安裝卡夫卡耗費的例子：

~ $ dcos package install kafka 
Installing Marathon app for package [kafka] version [1.1.11-0.10.0.0] 
Installing CLI subcommand for package [kafka] version [1.1.11-0.10.0.0] 
New command available: dcos kafka 
DC/OS Kafka Service is being installed. 

     Documentation: https://docs.mesosphere.com/usage/services/kafka/ 
     Issues: https://docs.mesosphere.com/support/ 
~ $ dcos kafka connection 
{ 
    "address": [ 
    "10.0.3.64:9951", 
    "10.0.3.68:9795", 
    "10.0.3.66:9531" 
    ], 
    "zookeeper": "master.mesos:2181/dcos-service-kafka", 
    "dns": [ 
    "broker-0.kafka.mesos:9951", 
    "broker-1.kafka.mesos:9795", 
    "broker-2.kafka.mesos:9531" 
    ], 
    "vip": "broker.kafka.l4lb.thisdcos.directory:9092" 
} 
~ $ dcos kafka topic create topic0 
{ 
    "message": "Output: Created topic \"topic0\".\n" 
} 
~ $ dcos node ssh --master-proxy --leader 
core@ip-10-0-7-56 ~ $ wget http://download.nextag.com/apache/kafka/0.10.0.1/kafka_2.11-0.10.0.1.tgz 
core@ip-10-0-7-56 ~ $ tar xf kafka_2.11-0.10.0.1.tgz 
core@ip-10-0-7-56 ~ $ cd kafka_2.11-0.10.0.1 
core@ip-10-0-7-56 ~/kafka_2.11-0.10.0.1 $ bin/kafka-console-producer.sh --broker-list broker.kafka.l4lb.thisdcos.directory:9092 --topic topic0 
This is a message 
This is another message 
^Ccore@ip-10-0-7-56 ~/kafka_2.11-0.10.0.1 $ bin/kafka-console-consumer.sh --zookeeper master.mesos:2181/dcos-service-kafka --topic topic0 --from-beginning 
This is a message 
This is another message 
^CProcessed a total of 2 messages 
$ bin/kafka-console-producer.sh --broker-list 10.0.3.64:9951,10.0.3.68:9795,10.0.3.66:9531 --topic topic0 
foo 
bar 
baz 
^Ccore@ip-10-0-7-56 ~/kafka_2.11-0.10.0.1 $ bin/kafka-console-consumer.sh --zookeeper master.mesos:2181/dcos-service-kafka --topic topic0 --from-beginning 
This is a message 
This is another message 
foo 
bar 
baz 
^CProcessed a total of 5 messages 

Answer 2

更新 - 這確實出現了已被丟失在/ etc/hosts中的條目引起的。我已經更新了我的terraform腳本以在安裝過程中編寫這些腳本，上面的示例現在按預期工作。

感謝您的幫助