Capacity Scheduler隊列問題

我正在嘗試在獨立節點中設置容量調度程序隊列，並且遇到問題。Capacity Scheduler隊列問題

我能夠通過UI在Cloudera Manager中創建可以看到它們的隊列，但是當我登錄並嘗試執行mapred queue -list時，它不顯示我設置的隊列。它只顯示我從配置中省略的默認隊列。

容量調度配置：

<?xml version="1.0"?> 
<configuration> 

<property> 
<name>yarn.scheduler.capacity.root.queues</name> 
<value>card,bank,digital</value> 
</property> 

<property> 
<name>yarn.scheduler.capacity.root.capacity</name> 
<value>100</value> 
</property> 


<property> 
<name>yarn.scheduler.capacity.root.digital.capacity</name> 
<value>41</value> 
</property> 

<property> 
<name>yarn.scheduler.capacity.root.card.capacity</name> 
<value>29</value> 
</property> 


<property> 
<name>yarn.scheduler.capacity.root.bank.capacity</name> 
<value>30</value> 
</property> 

<property> 
<name>yarn.scheduler.capacity.root.digital.acl_submit_applications</name> 
<value>cloudera</value> 
</property>

下面日誌

2014-02-18 08:09:54,940 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing card 
capacity = 0.29 [= (float) configuredCapacity/100 ] 
asboluteCapacity = 0.29 [= parentAbsoluteCapacity * capacity ] 
maxCapacity = 1.0 [= configuredMaxCapacity ] 
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity)/100 otherwise ] 
userLimit = 100 [= configuredUserLimit ] 
userLimitFactor = 1.0 [= configuredUserLimitFactor ] 
maxApplications = 2900 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)] 
maxApplicationsPerUser = 2900 [= (int)(maxApplications * (userLimit/100.0f) * userLimitFactor) ] 
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ] 
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] 
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit/100.0f) * userLimitFactor),1) ] 
usedCapacity = 0.0 [= usedResourcesMemory/(clusterResourceMemory * absoluteCapacity)] 
absoluteUsedCapacity = 0.0 [= usedResourcesMemory/clusterResourceMemory] 
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ] 
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory)/maximumAllocationMemory ] 
numContainers = 0 [= currentNumContainers ] 
state = RUNNING [= configuredState ] 
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS: [= configuredAcls ] 

2014-02-18 08:09:54,941 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: card: capacity=0.29, absoluteCapacity=0.29, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0 
2014-02-18 08:09:54,941 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing bank 
capacity = 0.3 [= (float) configuredCapacity/100 ] 
asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ] 
maxCapacity = 1.0 [= configuredMaxCapacity ] 
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity)/100 otherwise ] 
userLimit = 100 [= configuredUserLimit ] 
userLimitFactor = 1.0 [= configuredUserLimitFactor ] 
maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)] 
maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit/100.0f) * userLimitFactor) ] 
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ] 
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] 
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit/100.0f) * userLimitFactor),1) ] 
usedCapacity = 0.0 [= usedResourcesMemory/(clusterResourceMemory * absoluteCapacity)] 
absoluteUsedCapacity = 0.0 [= usedResourcesMemory/clusterResourceMemory] 
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ] 
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory)/maximumAllocationMemory ] 
numContainers = 0 [= currentNumContainers ] 
state = RUNNING [= configuredState ] 
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS: [= configuredAcls ] 

2014-02-18 08:09:54,941 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: bank: capacity=0.3, absoluteCapacity=0.3, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0 
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing digital 
capacity = 0.41 [= (float) configuredCapacity/100 ] 
asboluteCapacity = 0.41 [= parentAbsoluteCapacity * capacity ] 
maxCapacity = 1.0 [= configuredMaxCapacity ] 
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity)/100 otherwise ] 
userLimit = 100 [= configuredUserLimit ] 
userLimitFactor = 1.0 [= configuredUserLimitFactor ] 
maxApplications = 4100 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)] 
maxApplicationsPerUser = 4100 [= (int)(maxApplications * (userLimit/100.0f) * userLimitFactor) ] 
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ] 
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] 
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit/100.0f) * userLimitFactor),1) ] 
usedCapacity = 0.0 [= usedResourcesMemory/(clusterResourceMemory * absoluteCapacity)] 
absoluteUsedCapacity = 0.0 [= usedResourcesMemory/clusterResourceMemory] 
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ] 
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory)/maximumAllocationMemory ] 
numContainers = 0 [= currentNumContainers ] 
state = RUNNING [= configuredState ] 
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS:cloudera [= configuredAcls ] 

2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: digital: capacity=0.41, absoluteCapacity=0.41, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0 
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: root: numChildQueue= 3, capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>usedCapacity=0.0, numApps=0, numContainers=0 
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0, asboluteMaxCapacity=1.0, state=RUNNING, acls=ADMINISTER_QUEUE:*SUBMIT_APPLICATIONS:* 
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing card 
capacity = 0.29 [= (float) configuredCapacity/100 ] 
asboluteCapacity = 0.29 [= parentAbsoluteCapacity * capacity ] 
maxCapacity = 1.0 [= configuredMaxCapacity ] 
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity)/100 otherwise ] 
userLimit = 100 [= configuredUserLimit ] 
userLimitFactor = 1.0 [= configuredUserLimitFactor ] 
maxApplications = 2900 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)] 
maxApplicationsPerUser = 2900 [= (int)(maxApplications * (userLimit/100.0f) * userLimitFactor) ] 
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ] 
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] 
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit/100.0f) * userLimitFactor),1) ] 
usedCapacity = 0.0 [= usedResourcesMemory/(clusterResourceMemory * absoluteCapacity)] 
absoluteUsedCapacity = 0.0 [= usedResourcesMemory/clusterResourceMemory] 
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ] 
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory)/maximumAllocationMemory ] 
numContainers = 0 [= currentNumContainers ] 
state = RUNNING [= configuredState ] 
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS: [= configuredAcls ] 

2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root: re-configured queue: card: capacity=0.29, absoluteCapacity=0.29, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0 
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing bank 
capacity = 0.3 [= (float) configuredCapacity/100 ] 
asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ] 
maxCapacity = 1.0 [= configuredMaxCapacity ] 
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity)/100 otherwise ] 
userLimit = 100 [= configuredUserLimit ] 
userLimitFactor = 1.0 [= configuredUserLimitFactor ] 
maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)] 
maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit/100.0f) * userLimitFactor) ] 
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ] 
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] 
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit/100.0f) * userLimitFactor),1) ] 
usedCapacity = 0.0 [= usedResourcesMemory/(clusterResourceMemory * absoluteCapacity)] 
absoluteUsedCapacity = 0.0 [= usedResourcesMemory/clusterResourceMemory] 
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ] 
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory)/maximumAllocationMemory ] 
numContainers = 0 [= currentNumContainers ] 
state = RUNNING [= configuredState ] 
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS: [= configuredAcls ] 

2014-02-18 08:09:54,943 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root: re-configured queue: bank: capacity=0.3, absoluteCapacity=0.3, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0 
2014-02-18 08:09:54,943 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing digital 
capacity = 0.41 [= (float) configuredCapacity/100 ] 
asboluteCapacity = 0.41 [= parentAbsoluteCapacity * capacity ] 
maxCapacity = 1.0 [= configuredMaxCapacity ] 
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity)/100 otherwise ] 
userLimit = 100 [= configuredUserLimit ] 
userLimitFactor = 1.0 [= configuredUserLimitFactor ] 
maxApplications = 4100 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)] 
maxApplicationsPerUser = 4100 [= (int)(maxApplications * (userLimit/100.0f) * userLimitFactor) ] 
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ] 
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] 
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit/100.0f) * userLimitFactor),1) ] 
usedCapacity = 0.0 [= usedResourcesMemory/(clusterResourceMemory * absoluteCapacity)] 
absoluteUsedCapacity = 0.0 [= usedResourcesMemory/clusterResourceMemory] 
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ] 
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory)/maximumAllocationMemory ] 
numContainers = 0 [= currentNumContainers ] 
state = RUNNING [= configuredState ] 
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS:cloudera [= configuredAcls ] 

2014-02-18 08:09:54,943 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root: re-configured queue: digital: capacity=0.41, absoluteCapacity=0.41, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0 
2014-02-18 08:09:54,943 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=cloudera IP=127.0.0.1 OPERATION=refreshQueues TARGET=AdminService RESULT=SUCCESS 
2014-02-18 08:10:19,772 INFO org.apache.hadoop.yarn.server.resourcemanager.AdminService: RM Admin: refreshAdminAcls invoked by user cloudera 
2014-02-18 08:10:19,792 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=cloudera IP=127.0.0.1 OPERATION=refreshAdminAcls TARGET=AdminService RESULT=SUCCESS

Mapred命令輸出

Queue Name : default 
Queue State : running 
Scheduling Info : Queue configuration 
Capacity Percentage: 100.0% 
User Limit: 100% 
Priority Supported: NO 
------------- 
Map tasks 
Capacity: 2 slots 
Used capacity: 0 (0.0% of Capacity) 
Running tasks: 0 
------------- 
Reduce tasks 
Capacity: 2 slots 
Used capacity: 0 (0.0% of Capacity) 
Running tasks: 0 
------------- 
Job info 
Number of Waiting Jobs: 0 
Number of Initializing Jobs: 0 
Number of users who have submitted jobs: 0

來源

2014-02-18 user3315248

你還設置yarn.resourcemanager.scheduler.class參數org.apache.hadoop使用100％的資源。 yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler在conf/yarn-site.xml中？如果您在重新啓動後修改了隊列，我們需要運行「$ HADOOP_YARN_HOME/bin/yarn rmadmin -refreshQueues」來刷新隊列列表，您可能還需要重啓羣集才能使更改生效。 – rVr

謝謝rvr的回覆。我已經嘗試過這些，但仍然無法使用。 – user3315248

找出問題所在。根本原因是部署了客戶端配置，但不知何故/ etc/hadoop/conf未指向最新部署的配置。我改變了手動，它的工作 – user3315248

變化能力scheduler.xml下面的屬性在用戶隊列

yarn.scheduler.capacity.root.it.user-limit-factor=2 yarn.scheduler.capacity.root.price.user-limit-factor=1

來源

2017-08-03 07:33:08 dilshad

Capacity Scheduler隊列問題

回答

相關問題