2017-07-11 131 views
0

我想在本地機器上啓動一個hadoop單節點集羣。我已根據https://amodernstory.com/2014/09/23/installing-hadoop-on-mac-osx-yosemite/配置了以下文件:hadoop-env.sh,core-site.xml,mapred-site.xml和hdfs-site.xml。當我運行該腳本start-dfs.sh然後在命令jps我看到數據管理部已經啓動並運行(運行start-dfs.sh後右):如何將文件複製到HDFS?

15735 Jps 
15548 DataNode 
15660 SecondaryNameNode 
15453 NameNode 

幾秒鐘後,我重新運行該命令jps,我看到datanode沒有運行。爲什麼?如何解決這個問題?

之後,我運行腳本start-yarn.sh,然後運行命令jps。我看到:

15955 NodeManager 
16011 Jps 
15660 SecondaryNameNode 
15453 NameNode 
15854 ResourceManager 

我最終的目標是從我的本地文件系統將文件複製到HDFS。爲此,我運行命令hdfs dfs -copyFromLocal /source-file-path/filename /destination-file-path/。我收到以下錯誤:

17/07/10 17:09:00 WARN hdfs.DataStreamer: DataStreamer Exception 
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /pay/txnlinking/redshift.yml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. 
    at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1733) 
    at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:265) 
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2496) 
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:828) 
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:506) 
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) 
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447) 
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) 
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845) 
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:422) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) 
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455) 

    at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481) 
    at org.apache.hadoop.ipc.Client.call(Client.java:1427) 
    at org.apache.hadoop.ipc.Client.call(Client.java:1337) 
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) 
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) 
    at com.sun.proxy.$Proxy10.addBlock(Unknown Source) 
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:440) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335) 
    at com.sun.proxy.$Proxy11.addBlock(Unknown Source) 
    at org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1733) 
    at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1536) 
    at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:658) 
copyFromLocal: File /pay/txnlinking/redshift.yml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. 

如何避免上述錯誤並將文件複製到HDFS?

P.S:在複製之前,我明確在HDFS中創建了目標路徑文件夾。

+0

您是否檢查過datanode日誌? – Serhiy

+0

這個錯誤告訴你'文件只能被複制到0節點,這意味着你沒有運行datanode,hdfs沒有正確啓動。配置或網絡拓撲可能有問題(例如,namenode無法連接datanode)。首先檢查日誌。 – maxteneff

回答

1

首先刪除您在core-site.xml中指定的hadoop.tmp.dir文件夾的內容。然後使用hdfs namenode -format做一個namenode格式。您的datanode應該正常啓動並且正常運行,之後所有的複製操作都將成功執行。

+0

是你在tmp中默認的hadoop目錄,如果你應該改變它,否則你每次重啓時都會丟失hdfs – VIPER

+0

我認爲這是因爲我每次重啓時都會丟失HDFS中的文件。我盲目地使用了我在我的問題中共享的文章中指定的配置。但是現在我遵循[這個答案](https://stackoverflow.com/a/28379125/2531472)來避免這種情況發生。 –

1

hadoop namenode -format 

然後使用

stop-all.sh 

然後停止所有服務使用

start-all.sh 

start-all.sh和stop-all.sh已廢棄使用重新啓動所有服務start-dfs.sh和stop-dfs.sh代替

+0

不幸的是,只是格式化namenode並沒有完全解決。我能解決這個問題。看到我對這個問題的回答。非常感謝:) –

相關問題