Datanode和nodemanger出現在jps中，但沒有正確啓動

Question

當我執行jps命令datanode和nodemanagger出現，但它似乎沒有正確啓動，因爲如果我檢查日誌，似乎他們沒有正確運行。

時，我格式化NameNode和開始，我得到我在HDFS-site.xml中設置自動創建在namenodefolder一個版本的文件集羣名稱節點節點：

clusterID=CID-76572234-4ef7-4e6a-8ec5-1f54fe22b17d 
cTime=0 
storageType=NAME_NODE 
blockpoolID=BP-141486958-10.17.0.88-1463916426343 
layoutVersion=-63 

但在節點在哪裏的DataNode這文件未創建，我在hdfs-site.xml中設置的文件夾datanode被創建，但其內部爲空，我不知道它是否正常。

節點管理器日誌：

STARTUP_MSG: java = 1.8.0_91 
************************************************************/ 
2016-05-22 11:41:11,219 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT] 
2016-05-22 11:41:12,264 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher 
2016-05-22 11:41:12,265 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher 
2016-05-22 11:41:12,266 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService 
2016-05-22 11:41:12,266 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices 
2016-05-22 11:41:12,266 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl 
2016-05-22 11:41:12,267 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher 
2016-05-22 11:41:12,286 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl 
2016-05-22 11:41:12,286 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.NodeManager 
2016-05-22 11:41:12,326 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 
2016-05-22 11:41:12,397 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 
2016-05-22 11:41:12,398 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system started 
2016-05-22 11:41:12,420 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler 
2016-05-22 11:41:12,421 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.sharedcache.SharedCacheUploadEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.sharedcache.SharedCacheUploadService 
2016-05-22 11:41:12,421 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: per directory file limit = 8192 
2016-05-22 11:41:12,478 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: usercache path : file:/tmp/hadoop-hadoopadmin/nm-local-dir/usercache_DEL_1463913672424 
2016-05-22 11:41:12,529 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker 
2016-05-22 11:41:12,548 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@2dfaea86 
2016-05-22 11:41:12,548 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Using ResourceCalculatorProcessTree : null 
2016-05-22 11:41:12,549 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Physical memory check enabled: true 
2016-05-22 11:41:12,549 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Virtual memory check enabled: true 
2016-05-22 11:41:12,552 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: NodeManager configured with 8 G physical memory allocated to containers, which is more than 80% of the total physical memory available (3.9 G). Thrashing might happen. 
2016-05-22 11:41:12,557 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Initialized nodemanager for null: physical-memory=8192 virtual-memory=17204 virtual-cores=8 
2016-05-22 11:41:12,596 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 
2016-05-22 11:41:12,619 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 40484 
2016-05-22 11:41:12,651 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the server 
2016-05-22 11:41:12,651 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Blocking new container-requests as container manager rpc server is still starting. 
2016-05-22 11:41:12,651 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 
2016-05-22 11:41:12,652 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 40484: starting 
2016-05-22 11:41:12,661 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: Updating node address : ubuntuslave:40484 
2016-05-22 11:41:12,668 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 
2016-05-22 11:41:12,669 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8040 
2016-05-22 11:41:12,671 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the server 
2016-05-22 11:41:12,672 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8040: starting 
2016-05-22 11:41:12,672 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 
2016-05-22 11:41:12,673 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer started on port 8040 
2016-05-22 11:41:12,675 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: ContainerManager started at ubuntuslave/10.17.0.89:40484 
2016-05-22 11:41:12,675 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: ContainerManager bound to 0.0.0.0/0.0.0.0:0 
2016-05-22 11:41:12,676 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at 0.0.0.0:8042 
2016-05-22 11:41:12,749 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 
2016-05-22 11:41:12,758 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 
2016-05-22 11:41:12,763 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.nodemanager is not defined 
2016-05-22 11:41:12,771 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 
2016-05-22 11:41:12,773 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context node 
2016-05-22 11:41:12,773 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 
2016-05-22 11:41:12,773 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 
2016-05-22 11:41:12,776 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /node/* 
2016-05-22 11:41:12,777 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 
2016-05-22 11:41:12,786 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 8042 
2016-05-22 11:41:12,786 INFO org.mortbay.log: jetty-6.1.26 
2016-05-22 11:41:12,813 INFO org.mortbay.log: Extract jar:file:/usr/local/hadoop-2.7.1/share/hadoop/yarn/hadoop-yarn-common-2.7.1.jar!/webapps/node to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp 
2016-05-22 11:41:13,010 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8042 
2016-05-22 11:41:13,010 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /node started at 8042 
2016-05-22 11:41:13,316 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 
2016-05-22 11:41:13,324 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at masternode/10.18.0.50:8031 
2016-05-22 11:41:13,417 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out 0 NM container statuses: [] 
2016-05-22 11:41:13,426 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registering with RM using containers :[] 
2016-05-22 11:41:33,471 INFO org.apache.hadoop.ipc.Client: Retrying connect to server 

數據節點日誌：

STARTUP_MSG: java = 1.8.0_91 
************************************************************/ 
2016-05-22 11:40:40,852 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT] 
2016-05-22 11:40:41,523 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 
2016-05-22 11:40:41,607 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 
2016-05-22 11:40:41,607 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started 
2016-05-22 11:40:41,612 INFO org.apache.hadoop.hdfs.server.datanode.BlockScanner: Initialized block scanner with targetBytesPerSec 1048576 
2016-05-22 11:40:41,614 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname is ubuntuslave 
2016-05-22 11:40:41,620 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting DataNode with maxLockedMemory = 0 
2016-05-22 11:40:41,644 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming server at /0.0.0.0:50010 
2016-05-22 11:40:41,646 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s 
2016-05-22 11:40:41,646 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Number threads for balancing is 5 
2016-05-22 11:40:41,739 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 
2016-05-22 11:40:41,750 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 
2016-05-22 11:40:41,768 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.datanode is not defined 
2016-05-22 11:40:41,776 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 
2016-05-22 11:40:41,779 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode 
2016-05-22 11:40:41,780 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 
2016-05-22 11:40:41,780 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 
2016-05-22 11:40:41,796 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 52013 
2016-05-22 11:40:41,796 INFO org.mortbay.log: jetty-6.1.26 
2016-05-22 11:40:41,990 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:52013 
2016-05-22 11:40:42,109 INFO org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Listening HTTP traffic on /0.0.0.0:50075 
2016-05-22 11:40:42,298 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = hadoopadmin 
2016-05-22 11:40:42,298 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = supergroup 
2016-05-22 11:40:42,343 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 
2016-05-22 11:40:42,361 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020 
2016-05-22 11:40:42,388 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:50020 
2016-05-22 11:40:42,400 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null 
2016-05-22 11:40:42,424 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default> 
2016-05-22 11:40:42,436 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to masternode/10.18.0.50:9000 starting to offer service 
2016-05-22 11:40:42,444 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting 
2016-05-22 11:40:42,445 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 
2016-05-22 11:41:02,555 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: masternode/10.18.0.50:9000. Already tried 0 time(s); maxRetries=45 

紗-site.xml中：

<configuration> 
<property> 
<name>yarn.resourcemanager.resource-tracker.address</name> 
<value>masternode:8031</value> 
</property> 
<property> 
<name>yarn.resourcemanager.address</name> 
<value>masternode:8032</value> 
</property> 
<property> 
<name>yarn.resourcemanager.scheduler.address</name> 
<value>masternode:8030</value> 
</property> 
<property> 
<name>yarn.resourcemanager.admin.address</name> 
<value>masternode:8033</value> 
</property> 
<property> 
<name>yarn.resourcemanager.webapp.address</name> 
<value>masternode:8088</value> 
</property> 
</configuration> 

核心的site.xml：

<configuration> 
<property> 
<name>fs.defaultFS</name> 
<value>masternode:9000</value> 
</property> 
</configuration> 

個

HDFS-site.xml中：

<configuration> 
<property> 
<name>dfs.name.dir</name> 
<value>file:///home/hadoopadmin/hadooptmp</value> 
</property> 
<property> 
<name>dfs.data.dir</name> 
<value>file:///home/hadoopadmin/hadooptmp</value> 
</property> 
<property> 
<name>dfs.replication</name> 
<value>1</value> 
</property> 
</configuration> 

主人檔案：

masternode 

奴文件：

ubuntuslave 

你明白爲什麼它不工作？

Answer 1

查看datanode日誌後，它看起來像由於NameNode錯誤。看起來namenode已經關閉了。由於NameNode已關閉，Datanode無法正常啓動。

Here is what you can do : 
A. start the namenode. 
B. verify that namenode is running properly. 
C. start the datanode. verify it started properly. 
D. Run your spark application