檢索kerberized集羣中的狀態映射時出錯

HDP-2.5.3.0。檢索kerberized集羣中的狀態映射時出錯

自定義處理器使用State API來保存一些數據。

try { 
stateMap = stateManager.getState(Scope.CLUSTER); 
stateMapProperties = new HashMap<>(stateMap.toMap()); 
logger.debug("Retrieved the statemap : " + stateMapProperties); 
... 

... 

... 
} catch (IOException ioe) { 
logger.error("Couldn't load the state map", ioe); 
throw new ProcessException(ioe); 
}

處理器正常工作在我的本地機器的NiFi但是當我把它其中有2個NiFi節點我們的（採用Kerberos）開發羣集上時，出現以下錯誤（例外）：

java.io.IOException: Failed to obtain value from ZooKeeper for component with ID d7fff389-015a-1000-ffff-ffffd04d1279 with exception code NOAUTH 
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:420) ~[na:na] 
at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63) ~[na:na] 
at com.datalake.processors.SQLServerCDCProcessor.getDataFromChangeTables(SQLServerCDCProcessor.java:480) [nifi-NiFiCDCPoC-processors-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] 
at com.datalake.processors.SQLServerCDCProcessor.onTrigger(SQLServerCDCProcessor.java:191) [nifi-NiFiCDCPoC-processors-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] 
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.2.jar:1.1.2] 
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.2.jar:1.1.2] 
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.2.jar:1.1.2] 
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.2.jar:1.1.2] 
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.2.jar:1.1.2] 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_112] 
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_112] 
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_112] 
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_112] 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112] 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112] 
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] 
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /nifi/components/d7fff389-015a-1000-ffff-ffffd04d1279 
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113) ~[na:na] 
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[na:na] 
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) ~[na:na] 
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184) ~[na:na] 
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403) ~[na:na] 
. 
. 
. 
. 
. 
. 
. 
. 
. 
org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Failed to obtain value from ZooKeeper for component with ID d7fff389-015a-1000-ffff-ffffd04d1279 with exception code NOAUTH 
at com.datalake.processors.SQLServerCDCProcessor.getDataFromChangeTables(SQLServerCDCProcessor.java:493) ~[nifi-NiFiCDCPoC-processors-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] 
at com.datalake.processors.SQLServerCDCProcessor.onTrigger(SQLServerCDCProcessor.java:191) ~[nifi-NiFiCDCPoC-processors-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] 
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.2.jar:1.1.2] 
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.2.jar:1.1.2] 
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.2.jar:1.1.2] 
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.2.jar:1.1.2] 
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.2.jar:1.1.2] 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_112] 
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_112] 
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_112] 
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_112] 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112] 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112] 
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] 
Caused by: java.io.IOException: Failed to obtain value from ZooKeeper for component with ID d7fff389-015a-1000-ffff-ffffd04d1279 with exception code NOAUTH 
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:420) ~[na:na] 
at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63) ~[na:na] 
at com.datalake.processors.SQLServerCDCProcessor.getDataFromChangeTables(SQLServerCDCProcessor.java:480) ~[nifi-NiFiCDCPoC-processors-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] 
... 13 common frames omitted 
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /nifi/components/d7fff389-015a-1000-ffff-ffffd04d1279 
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113) ~[na:na] 
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[na:na] 
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) ~[na:na] 
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184) ~[na:na] 
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403) ~[na:na] 
... 15 common frames omitted

以下是在國家management.xml

<cluster-provider> 
     <id>zk-provider</id> 
     <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> 
     <property name="Connect String">l4373t.sss.se.scania.com:2181,l4283t.sss.se.scania.com:2181,l4284t.sss.se.scania.com:2181</property> 
     <property name="Root Node">/nifi</property> 
     <property name="Session Timeout">10 seconds</property> 
     <property name="Access Control">CreatorOnly</property> 
    </cluster-provider>

任何想法的條目？

*****編輯1 ***** 添加zk jaas配置。

bash-4.2$ cat zookeeper-jaas.conf 
Client { 
    com.sun.security.auth.module.Krb5LoginModule required 
    useKeyTab=true 
    keyTab="/usr/local/nifi/keys/nifi_l4513t.sss.se.com.keytab" 
    storeKey=true 
    useTicketCache=true 
    principal="nifi/[email protected]"; 
};

在bootstrap.conf文件中的條目（如 'java.arg.16'）：

bash-4.2$ vi bootstrap.conf 
# 

# Java command to use when running NiFi 
java=java 

# Username to use when running NiFi. This value will be ignored on Windows. 
run.as= 

# Configure where NiFi's lib and conf directories live 
lib.dir=./lib 
conf.dir=./conf 

# How long to wait after telling NiFi to shutdown before explicitly killing the Process 
graceful.shutdown.seconds=20 

# Disable JSR 199 so that we can use JSP's without running a JDK 
java.arg.1=-Dorg.apache.jasper.compiler.disablejsr199=true 

# JVM memory settings 
java.arg.2=-Xms1024m 
java.arg.3=-Xmx2048m 

# Enable Remote Debugging 
#java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 

java.arg.4=-Djava.net.preferIPv4Stack=true 

# allowRestrictedHeaders is required for Cluster/Node communications to work properly 
java.arg.5=-Dsun.net.http.allowRestrictedHeaders=true 
java.arg.6=-Djava.protocol.handler.pkgs=sun.net.www.protocol 

java.arg.7=-Dorg.apache.nifi.bootstrap.config.log.dir=/var/log/nifi 

# The G1GC is still considered experimental but has proven to be very advantageous in providing great 
# performance without significant "stop-the-world" delays. 
java.arg.13=-XX:+UseG1GC 

#Set headless mode by default 
java.arg.14=-Djava.awt.headless=true 

java.arg.15=-Djava.security.auth.login.config=/usr/local/nifi/conf/kafka-jaas.conf 
java.arg.16=-Djava.security.auth.login.config=/usr/local/nifi/conf/zookeeper-jaas.conf 
# Master key in hexadecimal format for encrypted sensitive configuration values 
nifi.bootstrap.sensitive.key= 

### 
# Notification Services for notifying interested parties when NiFi is stopped, started, dies 
###

*****編輯-2 *****提供了現有卡夫卡-jaas.conf

bash-4.2$ cat kafka-jaas.conf 
KafkaServer { 
    com.sun.security.auth.module.Krb5LoginModule required 
    useKeyTab=true 
    storeKey=true 
    renewTicket=true 
    useTicketCache=true 
    serviceName="kafka" 
    keyTab="/usr/local/nifi/keys/nifi_l4513t.sss.se.com.keytab" 
    principal="nifi/[email protected]"; 
}; 
KafkaClient { 
    com.sun.security.auth.module.Krb5LoginModule required 
    useKeyTab=true 
    useTicketCache=true 
    renewTicket=true 
    serviceName="kafka" 
    keyTab="/usr/local/nifi/keys/nifi_l4513t.sss.se.com.keytab" 
    principal="nifi/[email protected]"; 
}; 
Client { 
    com.sun.security.auth.module.Krb5LoginModule required 
    useKeyTab=true 
    storeKey=true 
    useTicketCache=true 
    serviceName="kafka" 
    keyTab="/usr/local/nifi/keys/nifi_l4513t.sss.se.com.keytab" 
    principal="nifi/[email protected]"; 
};

來源

2017-04-26 Kaliyug Antagonist

如果你跟一個基於Kerberos的ZooKeeper再有就是超越國家management.xml需要額外的配置。看看管理員指南部分上固定的ZooKeeper，特別是部分「Kerberizing NiFi的動物園管理員客戶端」：

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#zk_kerberos_client

編輯：

本文介紹的不同JAAS場景爲例：

https://community.hortonworks.com/content/kbentry/28180/how-to-configure-hdf-12-to-send-to-and-get-data-fr.html

大部分的文章示例使用嵌入式ZK，所以服用了這一點，我認爲你會需要這樣的東西：

Client { 
    com.sun.security.auth.module.Krb5LoginModule required 
    useKeyTab=true 
    keyTab="./conf/nifi.keytab" 
    storeKey=true 
    useTicketCache=false 
    principal="[email protected]」; 
}; 

KafkaClient { 
    com.sun.security.auth.module.Krb5LoginModule required 
    useTicketCache=true 
    renewTicket=true 
    serviceName="kafka" 
    useKeyTab=true 
    keyTab="./conf/nifi.keytab" 
    principal="[email protected]"; 
};

來源

2017-04-26 13:38:11

我看了那部分並做了修改，仍然錯誤依然存在。編輯原始問題以包含更改。 –

@KaliyugAntagonist好吧，我不確定您是否可以指定兩個JAAS文件（Kafka和ZK），我想您可能需要創建一個包含兩個條目的單個文件，然後在bootstrap.conf中只指定一個文件 –

如果這不起作用，您可能需要啓用Kerberos調試... java.arg.17 = -Dsun.security.krb5.debug = true並查看在nifi-boostrap.log中打印出的內容處理器。 –

檢索kerberized集羣中的狀態映射時出錯

回答

相關問題