如何運行的Hadoop節點管理器作爲特權帳戶的Windows

Question

我的工作在以下版本：

OS- Windows 8的

Hadoop- 2.3（單節點集羣），Pig- 0.12.1，Hive- 0.13

我可以使用Windows命令提示符（無管理員權限）啓動Namenode，Datanode，ResourceManager和NodeManager。當我嘗試運行mapreduce程序時，映射任務未啓動，它始終顯示爲「映射-0％」。

注： 如果我從Windows啓動「節點管理器」命令提示符具有管理員權限，然後MapReduce的程序工作正常。

節點管理器日誌：

14/10/07 14:44:18 INFO http.HttpServer2: adding path spec: /node/* 
14/10/07 14:44:18 INFO http.HttpServer2: adding path spec: /ws/* 
14/10/07 14:44:18 INFO http.HttpServer2: Jetty bound to port 8042 
14/10/07 14:44:18 INFO mortbay.log: jetty-6.1.26 
14/10/07 14:44:18 INFO mortbay.log: Extract jar:file:/C:/kaveen/BDSDK/1 
.1.0.8/SDK/Hadoop/share/hadoop/yarn/hadoop-yarn-common-2.3.0.jar!/webapps/node t 
o C:\Users\kaveen\AppData\Local\Temp\Jetty_localhost_8042_node____j463i9\webapp 
14/10/07 14:44:19 INFO mortbay.log: Started SelectChannelConnector@localhost:804 
2 
14/10/07 14:44:19 INFO webapp.WebApps: Web app /node started at 8042 
14/10/07 14:44:19 INFO webapp.WebApps: Registered webapp guice modules 
14/10/07 14:44:19 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0 
:8031 
14/10/07 14:44:19 INFO nodemanager.NodeStatusUpdaterImpl: Registering with RM us 
ing finished containers :[] 
14/10/07 14:44:19 INFO security.NMContainerTokenSecretManager: Rolling master-ke 
y for container-tokens, got key with id 1336526610 
14/10/07 14:44:19 INFO security.NMTokenSecretManagerInNM: Rolling master-key for 
nm-tokens, got key with id :-93940432 
14/10/07 14:44:19 INFO nodemanager.NodeStatusUpdaterImpl: Registered with Resour 
ceManager as 127.0.0.1:49344 with total resource of <memory:8192, vCores:8> 
14/10/07 14:44:19 INFO nodemanager.NodeStatusUpdaterImpl: Notifying ContainerMan 
ager to unblock new container-requests 

ResourceManager的日誌：

14/10/07 14:44:19 INFO util.RackResolver: Resolved 127.0.0.1 to /default-rack 
14/10/07 14:44:19 INFO resourcemanager.ResourceTrackerService: NodeManager from 
node 127.0.0.1(cmPort: 49344 httpPort: 8042) registered with capability: <memory 
:8192, vCores:8>, assigned nodeId 127.0.0.1:49344 
14/10/07 14:44:19 INFO rmnode.RMNodeImpl: 127.0.0.1:49344 Node Transitioned from 
NEW to RUNNING 
14/10/07 14:44:19 INFO capacity.CapacityScheduler: Added node 127.0.0.1:49344 cl 
usterResource: <memory:16384, vCores:16> 
14/10/07 14:44:19 INFO rmnode.RMNodeImpl: Node 127.0.0.1:49344 reported UNHEALTH 
Y with details: 1/1 local-dirs turned bad: /tmp/hadoop-kaveen/nm-local-dir;1/1 l 
og-dirs turned bad: C:/kaveen/BDSDK/1.1.0.8/SDK/Hadoop/logs/userlogs 
14/10/07 14:44:19 INFO rmnode.RMNodeImpl: 127.0.0.1:49344 Node Transitioned from 
RUNNING to UNHEALTHY 
14/10/07 14:44:19 INFO capacity.CapacityScheduler: Removed node 127.0.0.1:49344 
clusterResource: <memory:8192, vCores:8> 

在集羣節點"http://localhost:8088/cluster/nodes"看起來像下面sceenshots。

否創建節點。

不要任何人知道如何沒有管理員權限在Windows上運行「節點管理器」？

在此先感謝...

Answer 1

local-dirs turned bad: /tmp/hadoop-kaveen/nm-local-dir;1/1 
log-dirs turned bad: C:/kaveen/BDSDK/1.1.0.8/SDK/Hadoop/logs/userlogs 

你NM沒有一個有效的本地迪爾斯，你已經離開了那個顯然不打算在Windows上運行Linux的風格地方迪爾斯。日誌文件也是不正確的，NM可能沒有所需的權限。

絕對不需要提升NM。相反，固定本地迪爾斯紗線-site.xml中：

<property> 
<name>yarn.nodemanager.local-dirs</name> 
<value>C:/kaveen/BDSDK/1.1.0.8/SDK/Hadoop/local</value> 
</property> 

然後確保顯示目錄存在並且完全控制授予NM：

c:\>md C:\kaveen\BDSDK\1.1.0.8\SDK\Hadoop\local 
c:\>md C:\kaveen\BDSDK\1.1.0.8\SDK\Hadoop\logs 
c:\>cacls C:\kaveen\BDSDK\1.1.0.8\SDK\Hadoop\local /E /T /G <nm account>:F 
c:\>cacls C:\kaveen\BDSDK\1.1.0.8\SDK\Hadoop\logs/E /T /G <nm account>:F 

替換與相應的帳戶。

Answer 2

快速解決方法是在管理員權限下運行C:hadoop\sbin\start-dfs和C:hadoop\sbin\start-yarn。