2017-04-24 48 views
0

嘗試在Apache NiFi1.2.1上使用PutHDFS處理器時,使用下面的配置;PutHDFS on Apache NiFi錯誤

hadoop configuration reource : /usr/local/hadoop-2.7.0/etc/hadoop/core-site.xml, /usr/local/hadoop-2.7.0/etc/hadoop/hdfs-site.xml 
directory: /mydir 

我遇到了以下錯誤。

Caused by: org.apache.hadoop.ipc.RemoteException: File /tweets/.381623121831518.json could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. 
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550) 
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3067) 
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:722) 
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) 
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) 
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) 
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) 
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) 
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:422) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) 
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) 

回答

0

分辨率

我跟着以下步驟來糾正這個問題;

停止所有服務

$ cd $HADOOP_HOME 
$ sbin/stop-all.sh 

在HDFS-site.xml中提到

$rm datanode 
$rm namenode 

格式的NameNode

hadoop namenode -format 

啓動所有的Hadoop服務

刪除NameNode和數據節點的目錄

考證:

檢查所有服務運行在PutHDFS處理器指定的在/ MYDIR轉移

bash-3.2# jps 
61488 ResourceManager 
57128 RunNiFi 
61160 NameNode 
61256 DataNode 
57129 NiFi 
61609 Jps 
61371 SecondaryNameNode 
61582 NodeManager 

檢查文件 - >目標目錄。應在此目錄中傳輸文件

$ hdfs dfs -ls /mydir