YARN MapReduce沒有足夠的內存

Question

我想在Yarn MapReduce中啓動我的應用程序。我在4臺主機（1個資源管理器和4個節點管理器）中設置了MapReduce。每個主機都有2個內核和4GB的RAM。

當我運行我的應用程序時，由於缺少RAM [1]而中止。我應該如何設置Yarn MapReduce以便作業不會耗盡RAM？

[1]我有錯。

2016-02-09 16:01:41,607 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147 
_0001_01_000001: 289.6 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used 
2016-02-09 16:01:44,612 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147 
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used 
2016-02-09 16:01:47,616 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147 
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used 
2016-02-09 16:01:50,621 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147 
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used 
2016-02-09 16:01:53,626 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147 
_0001_01_000001: 291.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used 
2016-02-09 16:01:56,631 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147 
_0001_01_000001: 295.1 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used 
2016-02-09 16:01:59,643 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147 
_0001_01_000001: 295.3 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used 
(...) 
2016-02-09 16:02:01,173 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/syslog 
2016-02-09 16:02:01,174 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/stdout 
2016-02-09 16:02:01,174 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/stderr 
2016-02-09 16:02:01,250 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001 
2016-02-09 16:02:02,644 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455032058147_0001_01_000001 
2016-02-09 16:03:10,146 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 
2016-02-09 16:03:11,147 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 
2016-02-09 16:03:12,148 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 
2016-02-09 16:03:13,149 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 

Answer 1

您需要配置yarn-site.xml給紗的內存量和它的容器：

<property> 
    <name>yarn.nodemanager.resource.memory-mb</name> 
    <value>4096</value> 
</property> 
<property> 
    <name>yarn.scheduler.minimum-allocation-mb</name> 
    <value>1024</value> 
</property> 

入住此示例yarn-site.xml文件。在運行MapReduce作業時，您還需要在mapred-site.xml文件中設置類似的屬性。本文可能會對您有所幫助 - link。