You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Munna <mu...@gmail.com> on 2013/11/27 18:42:59 UTC
Capacity Scheduler Issue
Hi Flocks,
Since, last two days I am about to configure Capacity Scheduler. Here, how
I have struggling L….
I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
Queue’s as per the Capacity Scheduler Documentation and those queues shown
in RM UI.
After configuration I tried to run Jobs, Cap Scheduler not identified that
queue’s. where I have check queues list with “mapred queue –list”, which
showing all configured Q’s.
I wrote a mail’s to groups for solution, Mr.Olivier has been given some
idea about that, based on his views I dig more.
After I went to all the RM log, Cap Scheduler initiating only default
“default”, I have tested with *default queue* it works for me. And I have
created one more queue called “dev”, in this Queue User unable to run the
jobs and its unable to identifying users Queue.
I have attached Cap Scheduler configuration file for your information. Some
O/P for ur information.
*[user@host ~]$ mapred queue -list*
*13/11/27 09:26:38 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
*13/11/27 09:26:38 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
*======================*
*Queue Name : dev*
*Queue State : running*
*Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
CurrentCapacity: 0.0*
*======================*
*Queue Name : default*
*Queue State : running*
*Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
0.0*
*RM log Scheduler loading info:*
2013-11-27 08:54:58,521 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
ADMINISTER_QUEUE:
2013-11-27 08:54:58,521 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Initialized parent-queue root name=root, fullname=root
2013-11-27 08:54:58,534 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
*Initializing
default*
capacity = 0.7 [= (float) configuredCapacity / 100 ]
asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
(parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
(int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
(userLimit / 100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
minimumAllocationMemory) / maximumAllocationMemory ]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
minimumAllocationMemory) / maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [= configuredAcls
]
2013-11-27 08:54:58,534 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
usedResources=<memory:0, vCores:0>usedCapacity=0.0,
absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2013-11-27 08:54:58,543 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
*Initializing
dev*
capacity = 0.3 [= (float) configuredCapacity / 100 ]
asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 0.5 [= configuredMaxCapacity ]
absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
(parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
(int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
(userLimit / 100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
minimumAllocationMemory) / maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls ]
Can you guys please confirm, did I miss anything on configurations part or
is there any bug persist on 2.0.0?
Thanks
Munna
Re: Capacity Scheduler Issue
Posted by Munna <mu...@gmail.com>.
Hi All,
Thank you for taking your time to help me.
Now I can able to run the jobs within configured Queues.
Regards,
Munna
On Fri, Nov 29, 2013 at 2:22 AM, Jian He <jh...@hortonworks.com> wrote:
> Did the application get accepted ? If the application gets accepted but
> just not able to run, as I said, try increasing yarn.scheduler.capacity.maximum-am-resource-percent
> to a much larger value, maybe 0.8, just for troubleshoot. How large it
> should be depends on your requirements.
>
> yarn.scheduler.capacity.maximum-am-resource-percent controls the max
> number of concurrently running AMs.
> By that I mean the max memory allowed for allocating AM (defined by this
> property) divided by per AM memory usage , equals to the max number of
> concurrently running AMs.
>
> I have seen both of your queues only allow 1 active application.
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> Jian
>
>
> On Thu, Nov 28, 2013 at 12:23 PM, Adam Kawa <ka...@gmail.com> wrote:
>
>> I see that you have different settings for ACL:
>>
>> <name>yarn.scheduler.capacity.root.*default*
>> .acl_submit_applications</name><value>*yarn,mapred*
>> </value></property><property>
>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>> configuredAcls ]
>>
>> <name>yarn.scheduler.capacity.root.*dev*
>> .acl_submit_applications</name><value>*smunnavar,test*</value></property>
>>
>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls
>> ]
>>
>> Do you submit your jobs to dev queue using good user accounts? Did you
>> include a right config xml in a previous post? (xml says *smunnavar,test*,
>> but RM loads user,test).
>>
>> As a first troubleshooting steps, could you disable ACL for dev queue to
>> check if you can submit a job there?
>>
>> <property>
>> <name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
>> <value>*</value>
>> </property>
>>
>>
>>
>> 2013/11/28 Munna <mu...@gmail.com>
>>
>>> Hi,
>>>
>>>
>>> I think there is no solution on above issue, so i'll move to fair
>>> scheduler.
>>>
>>> Thanks to all...
>>>
>>> Munna
>>>
>>>
>>> On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
>>>
>>>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>>>
>>>>
>>>> what is the best value?
>>>>
>>>> Tx,
>>>> Munna
>>>>
>>>>
>>>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>>>>
>>>>> The log shows the both queues are properly picked up by the RM.
>>>>> If the problem is that your submitted application is not able to run,
>>>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>>>> this controls the max number of concurrently running AMs in the cluster.
>>>>>
>>>>> Jian
>>>>>
>>>>>
>>>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>>>>
>>>>>> Hi Flocks,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Since, last two days I am about to configure Capacity Scheduler.
>>>>>> Here, how I have struggling L….
>>>>>>
>>>>>>
>>>>>>
>>>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have
>>>>>> created 4 Queue’s as per the Capacity Scheduler Documentation and those
>>>>>> queues shown in RM UI.
>>>>>>
>>>>>>
>>>>>>
>>>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>>>> which showing all configured Q’s.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given
>>>>>> some idea about that, based on his views I dig more.
>>>>>>
>>>>>>
>>>>>>
>>>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>>>> “default”, I have tested with *default queue* it works for me. And I
>>>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>>>> the jobs and its unable to identifying users Queue.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have attached Cap Scheduler configuration file for your
>>>>>> information. Some O/P for ur information.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *[user@host ~]$ mapred queue -list*
>>>>>>
>>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>>>
>>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>>>
>>>>>> *======================*
>>>>>>
>>>>>> *Queue Name : dev*
>>>>>>
>>>>>> *Queue State : running*
>>>>>>
>>>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>>>> CurrentCapacity: 0.0*
>>>>>>
>>>>>> *======================*
>>>>>>
>>>>>> *Queue Name : default*
>>>>>>
>>>>>> *Queue State : running*
>>>>>>
>>>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>>>> CurrentCapacity: 0.0*
>>>>>>
>>>>>>
>>>>>>
>>>>>> *RM log Scheduler loading info:*
>>>>>>
>>>>>> 2013-11-27 08:54:58,521 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>>>> ADMINISTER_QUEUE:
>>>>>>
>>>>>> 2013-11-27 08:54:58,521 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>>> Initialized parent-queue root name=root, fullname=root
>>>>>>
>>>>>> 2013-11-27 08:54:58,534 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>>> default*
>>>>>>
>>>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>>>
>>>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>>>
>>>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>>>
>>>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>>
>>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>>
>>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>>
>>>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>>
>>>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>>>> 100.0f) * userLimitFactor) ]
>>>>>>
>>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>>> ]
>>>>>>
>>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory
>>>>>> / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>>
>>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>>
>>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>>> absoluteCapacity)]
>>>>>>
>>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>>> clusterResourceMemory]
>>>>>>
>>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>>> configuredMaximumAMResourcePercent ]
>>>>>>
>>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>>
>>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>>> configuredMaximumAMResourcePercent ]
>>>>>>
>>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>>
>>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>>
>>>>>> state = RUNNING [= configuredState ]
>>>>>>
>>>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>>>> configuredAcls ]
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013-11-27 08:54:58,534 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>>>
>>>>>> 2013-11-27 08:54:58,543 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>>> dev*
>>>>>>
>>>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>>>
>>>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>>>
>>>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>>>
>>>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>>
>>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>>
>>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>>
>>>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>>
>>>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>>>> 100.0f) * userLimitFactor) ]
>>>>>>
>>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>>> ]
>>>>>>
>>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory
>>>>>> / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>>
>>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>>
>>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>>> absoluteCapacity)]
>>>>>>
>>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>>> clusterResourceMemory]
>>>>>>
>>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>>> configuredMaximumAMResourcePercent ]
>>>>>>
>>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>>
>>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>>
>>>>>> state = RUNNING [= configuredState ]
>>>>>>
>>>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>>>> configuredAcls ]
>>>>>>
>>>>>>
>>>>>>
>>>>>> Can you guys please confirm, did I miss anything on configurations
>>>>>> part or is there any bug persist on 2.0.0?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Munna
>>>>>>
>>>>>
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Regards*
>>>>
>>>> *Munna*
>>>>
>>>
>>>
>>>
>>> --
>>> *Regards*
>>>
>>> *Munna*
>>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
*Regards*
*Munna*
Re: Capacity Scheduler Issue
Posted by Munna <mu...@gmail.com>.
Hi All,
Thank you for taking your time to help me.
Now I can able to run the jobs within configured Queues.
Regards,
Munna
On Fri, Nov 29, 2013 at 2:22 AM, Jian He <jh...@hortonworks.com> wrote:
> Did the application get accepted ? If the application gets accepted but
> just not able to run, as I said, try increasing yarn.scheduler.capacity.maximum-am-resource-percent
> to a much larger value, maybe 0.8, just for troubleshoot. How large it
> should be depends on your requirements.
>
> yarn.scheduler.capacity.maximum-am-resource-percent controls the max
> number of concurrently running AMs.
> By that I mean the max memory allowed for allocating AM (defined by this
> property) divided by per AM memory usage , equals to the max number of
> concurrently running AMs.
>
> I have seen both of your queues only allow 1 active application.
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> Jian
>
>
> On Thu, Nov 28, 2013 at 12:23 PM, Adam Kawa <ka...@gmail.com> wrote:
>
>> I see that you have different settings for ACL:
>>
>> <name>yarn.scheduler.capacity.root.*default*
>> .acl_submit_applications</name><value>*yarn,mapred*
>> </value></property><property>
>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>> configuredAcls ]
>>
>> <name>yarn.scheduler.capacity.root.*dev*
>> .acl_submit_applications</name><value>*smunnavar,test*</value></property>
>>
>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls
>> ]
>>
>> Do you submit your jobs to dev queue using good user accounts? Did you
>> include a right config xml in a previous post? (xml says *smunnavar,test*,
>> but RM loads user,test).
>>
>> As a first troubleshooting steps, could you disable ACL for dev queue to
>> check if you can submit a job there?
>>
>> <property>
>> <name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
>> <value>*</value>
>> </property>
>>
>>
>>
>> 2013/11/28 Munna <mu...@gmail.com>
>>
>>> Hi,
>>>
>>>
>>> I think there is no solution on above issue, so i'll move to fair
>>> scheduler.
>>>
>>> Thanks to all...
>>>
>>> Munna
>>>
>>>
>>> On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
>>>
>>>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>>>
>>>>
>>>> what is the best value?
>>>>
>>>> Tx,
>>>> Munna
>>>>
>>>>
>>>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>>>>
>>>>> The log shows the both queues are properly picked up by the RM.
>>>>> If the problem is that your submitted application is not able to run,
>>>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>>>> this controls the max number of concurrently running AMs in the cluster.
>>>>>
>>>>> Jian
>>>>>
>>>>>
>>>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>>>>
>>>>>> Hi Flocks,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Since, last two days I am about to configure Capacity Scheduler.
>>>>>> Here, how I have struggling L….
>>>>>>
>>>>>>
>>>>>>
>>>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have
>>>>>> created 4 Queue’s as per the Capacity Scheduler Documentation and those
>>>>>> queues shown in RM UI.
>>>>>>
>>>>>>
>>>>>>
>>>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>>>> which showing all configured Q’s.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given
>>>>>> some idea about that, based on his views I dig more.
>>>>>>
>>>>>>
>>>>>>
>>>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>>>> “default”, I have tested with *default queue* it works for me. And I
>>>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>>>> the jobs and its unable to identifying users Queue.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have attached Cap Scheduler configuration file for your
>>>>>> information. Some O/P for ur information.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *[user@host ~]$ mapred queue -list*
>>>>>>
>>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>>>
>>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>>>
>>>>>> *======================*
>>>>>>
>>>>>> *Queue Name : dev*
>>>>>>
>>>>>> *Queue State : running*
>>>>>>
>>>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>>>> CurrentCapacity: 0.0*
>>>>>>
>>>>>> *======================*
>>>>>>
>>>>>> *Queue Name : default*
>>>>>>
>>>>>> *Queue State : running*
>>>>>>
>>>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>>>> CurrentCapacity: 0.0*
>>>>>>
>>>>>>
>>>>>>
>>>>>> *RM log Scheduler loading info:*
>>>>>>
>>>>>> 2013-11-27 08:54:58,521 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>>>> ADMINISTER_QUEUE:
>>>>>>
>>>>>> 2013-11-27 08:54:58,521 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>>> Initialized parent-queue root name=root, fullname=root
>>>>>>
>>>>>> 2013-11-27 08:54:58,534 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>>> default*
>>>>>>
>>>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>>>
>>>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>>>
>>>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>>>
>>>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>>
>>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>>
>>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>>
>>>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>>
>>>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>>>> 100.0f) * userLimitFactor) ]
>>>>>>
>>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>>> ]
>>>>>>
>>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory
>>>>>> / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>>
>>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>>
>>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>>> absoluteCapacity)]
>>>>>>
>>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>>> clusterResourceMemory]
>>>>>>
>>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>>> configuredMaximumAMResourcePercent ]
>>>>>>
>>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>>
>>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>>> configuredMaximumAMResourcePercent ]
>>>>>>
>>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>>
>>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>>
>>>>>> state = RUNNING [= configuredState ]
>>>>>>
>>>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>>>> configuredAcls ]
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013-11-27 08:54:58,534 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>>>
>>>>>> 2013-11-27 08:54:58,543 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>>> dev*
>>>>>>
>>>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>>>
>>>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>>>
>>>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>>>
>>>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>>
>>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>>
>>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>>
>>>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>>
>>>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>>>> 100.0f) * userLimitFactor) ]
>>>>>>
>>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>>> ]
>>>>>>
>>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory
>>>>>> / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>>
>>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>>
>>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>>> absoluteCapacity)]
>>>>>>
>>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>>> clusterResourceMemory]
>>>>>>
>>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>>> configuredMaximumAMResourcePercent ]
>>>>>>
>>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>>
>>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>>
>>>>>> state = RUNNING [= configuredState ]
>>>>>>
>>>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>>>> configuredAcls ]
>>>>>>
>>>>>>
>>>>>>
>>>>>> Can you guys please confirm, did I miss anything on configurations
>>>>>> part or is there any bug persist on 2.0.0?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Munna
>>>>>>
>>>>>
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Regards*
>>>>
>>>> *Munna*
>>>>
>>>
>>>
>>>
>>> --
>>> *Regards*
>>>
>>> *Munna*
>>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
*Regards*
*Munna*
Re: Capacity Scheduler Issue
Posted by Munna <mu...@gmail.com>.
Hi All,
Thank you for taking your time to help me.
Now I can able to run the jobs within configured Queues.
Regards,
Munna
On Fri, Nov 29, 2013 at 2:22 AM, Jian He <jh...@hortonworks.com> wrote:
> Did the application get accepted ? If the application gets accepted but
> just not able to run, as I said, try increasing yarn.scheduler.capacity.maximum-am-resource-percent
> to a much larger value, maybe 0.8, just for troubleshoot. How large it
> should be depends on your requirements.
>
> yarn.scheduler.capacity.maximum-am-resource-percent controls the max
> number of concurrently running AMs.
> By that I mean the max memory allowed for allocating AM (defined by this
> property) divided by per AM memory usage , equals to the max number of
> concurrently running AMs.
>
> I have seen both of your queues only allow 1 active application.
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> Jian
>
>
> On Thu, Nov 28, 2013 at 12:23 PM, Adam Kawa <ka...@gmail.com> wrote:
>
>> I see that you have different settings for ACL:
>>
>> <name>yarn.scheduler.capacity.root.*default*
>> .acl_submit_applications</name><value>*yarn,mapred*
>> </value></property><property>
>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>> configuredAcls ]
>>
>> <name>yarn.scheduler.capacity.root.*dev*
>> .acl_submit_applications</name><value>*smunnavar,test*</value></property>
>>
>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls
>> ]
>>
>> Do you submit your jobs to dev queue using good user accounts? Did you
>> include a right config xml in a previous post? (xml says *smunnavar,test*,
>> but RM loads user,test).
>>
>> As a first troubleshooting steps, could you disable ACL for dev queue to
>> check if you can submit a job there?
>>
>> <property>
>> <name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
>> <value>*</value>
>> </property>
>>
>>
>>
>> 2013/11/28 Munna <mu...@gmail.com>
>>
>>> Hi,
>>>
>>>
>>> I think there is no solution on above issue, so i'll move to fair
>>> scheduler.
>>>
>>> Thanks to all...
>>>
>>> Munna
>>>
>>>
>>> On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
>>>
>>>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>>>
>>>>
>>>> what is the best value?
>>>>
>>>> Tx,
>>>> Munna
>>>>
>>>>
>>>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>>>>
>>>>> The log shows the both queues are properly picked up by the RM.
>>>>> If the problem is that your submitted application is not able to run,
>>>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>>>> this controls the max number of concurrently running AMs in the cluster.
>>>>>
>>>>> Jian
>>>>>
>>>>>
>>>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>>>>
>>>>>> Hi Flocks,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Since, last two days I am about to configure Capacity Scheduler.
>>>>>> Here, how I have struggling L….
>>>>>>
>>>>>>
>>>>>>
>>>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have
>>>>>> created 4 Queue’s as per the Capacity Scheduler Documentation and those
>>>>>> queues shown in RM UI.
>>>>>>
>>>>>>
>>>>>>
>>>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>>>> which showing all configured Q’s.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given
>>>>>> some idea about that, based on his views I dig more.
>>>>>>
>>>>>>
>>>>>>
>>>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>>>> “default”, I have tested with *default queue* it works for me. And I
>>>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>>>> the jobs and its unable to identifying users Queue.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have attached Cap Scheduler configuration file for your
>>>>>> information. Some O/P for ur information.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *[user@host ~]$ mapred queue -list*
>>>>>>
>>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>>>
>>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>>>
>>>>>> *======================*
>>>>>>
>>>>>> *Queue Name : dev*
>>>>>>
>>>>>> *Queue State : running*
>>>>>>
>>>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>>>> CurrentCapacity: 0.0*
>>>>>>
>>>>>> *======================*
>>>>>>
>>>>>> *Queue Name : default*
>>>>>>
>>>>>> *Queue State : running*
>>>>>>
>>>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>>>> CurrentCapacity: 0.0*
>>>>>>
>>>>>>
>>>>>>
>>>>>> *RM log Scheduler loading info:*
>>>>>>
>>>>>> 2013-11-27 08:54:58,521 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>>>> ADMINISTER_QUEUE:
>>>>>>
>>>>>> 2013-11-27 08:54:58,521 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>>> Initialized parent-queue root name=root, fullname=root
>>>>>>
>>>>>> 2013-11-27 08:54:58,534 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>>> default*
>>>>>>
>>>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>>>
>>>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>>>
>>>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>>>
>>>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>>
>>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>>
>>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>>
>>>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>>
>>>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>>>> 100.0f) * userLimitFactor) ]
>>>>>>
>>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>>> ]
>>>>>>
>>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory
>>>>>> / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>>
>>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>>
>>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>>> absoluteCapacity)]
>>>>>>
>>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>>> clusterResourceMemory]
>>>>>>
>>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>>> configuredMaximumAMResourcePercent ]
>>>>>>
>>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>>
>>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>>> configuredMaximumAMResourcePercent ]
>>>>>>
>>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>>
>>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>>
>>>>>> state = RUNNING [= configuredState ]
>>>>>>
>>>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>>>> configuredAcls ]
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013-11-27 08:54:58,534 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>>>
>>>>>> 2013-11-27 08:54:58,543 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>>> dev*
>>>>>>
>>>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>>>
>>>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>>>
>>>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>>>
>>>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>>
>>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>>
>>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>>
>>>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>>
>>>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>>>> 100.0f) * userLimitFactor) ]
>>>>>>
>>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>>> ]
>>>>>>
>>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory
>>>>>> / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>>
>>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>>
>>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>>> absoluteCapacity)]
>>>>>>
>>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>>> clusterResourceMemory]
>>>>>>
>>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>>> configuredMaximumAMResourcePercent ]
>>>>>>
>>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>>
>>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>>
>>>>>> state = RUNNING [= configuredState ]
>>>>>>
>>>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>>>> configuredAcls ]
>>>>>>
>>>>>>
>>>>>>
>>>>>> Can you guys please confirm, did I miss anything on configurations
>>>>>> part or is there any bug persist on 2.0.0?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Munna
>>>>>>
>>>>>
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Regards*
>>>>
>>>> *Munna*
>>>>
>>>
>>>
>>>
>>> --
>>> *Regards*
>>>
>>> *Munna*
>>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
*Regards*
*Munna*
Re: Capacity Scheduler Issue
Posted by Munna <mu...@gmail.com>.
Hi All,
Thank you for taking your time to help me.
Now I can able to run the jobs within configured Queues.
Regards,
Munna
On Fri, Nov 29, 2013 at 2:22 AM, Jian He <jh...@hortonworks.com> wrote:
> Did the application get accepted ? If the application gets accepted but
> just not able to run, as I said, try increasing yarn.scheduler.capacity.maximum-am-resource-percent
> to a much larger value, maybe 0.8, just for troubleshoot. How large it
> should be depends on your requirements.
>
> yarn.scheduler.capacity.maximum-am-resource-percent controls the max
> number of concurrently running AMs.
> By that I mean the max memory allowed for allocating AM (defined by this
> property) divided by per AM memory usage , equals to the max number of
> concurrently running AMs.
>
> I have seen both of your queues only allow 1 active application.
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> Jian
>
>
> On Thu, Nov 28, 2013 at 12:23 PM, Adam Kawa <ka...@gmail.com> wrote:
>
>> I see that you have different settings for ACL:
>>
>> <name>yarn.scheduler.capacity.root.*default*
>> .acl_submit_applications</name><value>*yarn,mapred*
>> </value></property><property>
>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>> configuredAcls ]
>>
>> <name>yarn.scheduler.capacity.root.*dev*
>> .acl_submit_applications</name><value>*smunnavar,test*</value></property>
>>
>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls
>> ]
>>
>> Do you submit your jobs to dev queue using good user accounts? Did you
>> include a right config xml in a previous post? (xml says *smunnavar,test*,
>> but RM loads user,test).
>>
>> As a first troubleshooting steps, could you disable ACL for dev queue to
>> check if you can submit a job there?
>>
>> <property>
>> <name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
>> <value>*</value>
>> </property>
>>
>>
>>
>> 2013/11/28 Munna <mu...@gmail.com>
>>
>>> Hi,
>>>
>>>
>>> I think there is no solution on above issue, so i'll move to fair
>>> scheduler.
>>>
>>> Thanks to all...
>>>
>>> Munna
>>>
>>>
>>> On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
>>>
>>>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>>>
>>>>
>>>> what is the best value?
>>>>
>>>> Tx,
>>>> Munna
>>>>
>>>>
>>>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>>>>
>>>>> The log shows the both queues are properly picked up by the RM.
>>>>> If the problem is that your submitted application is not able to run,
>>>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>>>> this controls the max number of concurrently running AMs in the cluster.
>>>>>
>>>>> Jian
>>>>>
>>>>>
>>>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>>>>
>>>>>> Hi Flocks,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Since, last two days I am about to configure Capacity Scheduler.
>>>>>> Here, how I have struggling L….
>>>>>>
>>>>>>
>>>>>>
>>>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have
>>>>>> created 4 Queue’s as per the Capacity Scheduler Documentation and those
>>>>>> queues shown in RM UI.
>>>>>>
>>>>>>
>>>>>>
>>>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>>>> which showing all configured Q’s.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given
>>>>>> some idea about that, based on his views I dig more.
>>>>>>
>>>>>>
>>>>>>
>>>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>>>> “default”, I have tested with *default queue* it works for me. And I
>>>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>>>> the jobs and its unable to identifying users Queue.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have attached Cap Scheduler configuration file for your
>>>>>> information. Some O/P for ur information.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *[user@host ~]$ mapred queue -list*
>>>>>>
>>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>>>
>>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>>>
>>>>>> *======================*
>>>>>>
>>>>>> *Queue Name : dev*
>>>>>>
>>>>>> *Queue State : running*
>>>>>>
>>>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>>>> CurrentCapacity: 0.0*
>>>>>>
>>>>>> *======================*
>>>>>>
>>>>>> *Queue Name : default*
>>>>>>
>>>>>> *Queue State : running*
>>>>>>
>>>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>>>> CurrentCapacity: 0.0*
>>>>>>
>>>>>>
>>>>>>
>>>>>> *RM log Scheduler loading info:*
>>>>>>
>>>>>> 2013-11-27 08:54:58,521 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>>>> ADMINISTER_QUEUE:
>>>>>>
>>>>>> 2013-11-27 08:54:58,521 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>>> Initialized parent-queue root name=root, fullname=root
>>>>>>
>>>>>> 2013-11-27 08:54:58,534 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>>> default*
>>>>>>
>>>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>>>
>>>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>>>
>>>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>>>
>>>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>>
>>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>>
>>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>>
>>>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>>
>>>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>>>> 100.0f) * userLimitFactor) ]
>>>>>>
>>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>>> ]
>>>>>>
>>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory
>>>>>> / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>>
>>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>>
>>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>>> absoluteCapacity)]
>>>>>>
>>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>>> clusterResourceMemory]
>>>>>>
>>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>>> configuredMaximumAMResourcePercent ]
>>>>>>
>>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>>
>>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>>> configuredMaximumAMResourcePercent ]
>>>>>>
>>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>>
>>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>>
>>>>>> state = RUNNING [= configuredState ]
>>>>>>
>>>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>>>> configuredAcls ]
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013-11-27 08:54:58,534 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>>>
>>>>>> 2013-11-27 08:54:58,543 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>>> dev*
>>>>>>
>>>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>>>
>>>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>>>
>>>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>>>
>>>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>>
>>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>>
>>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>>
>>>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>>
>>>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>>>> 100.0f) * userLimitFactor) ]
>>>>>>
>>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>>> ]
>>>>>>
>>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory
>>>>>> / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>>
>>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>>
>>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>>> absoluteCapacity)]
>>>>>>
>>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>>> clusterResourceMemory]
>>>>>>
>>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>>> configuredMaximumAMResourcePercent ]
>>>>>>
>>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>>
>>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>>
>>>>>> state = RUNNING [= configuredState ]
>>>>>>
>>>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>>>> configuredAcls ]
>>>>>>
>>>>>>
>>>>>>
>>>>>> Can you guys please confirm, did I miss anything on configurations
>>>>>> part or is there any bug persist on 2.0.0?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Munna
>>>>>>
>>>>>
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Regards*
>>>>
>>>> *Munna*
>>>>
>>>
>>>
>>>
>>> --
>>> *Regards*
>>>
>>> *Munna*
>>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
*Regards*
*Munna*
Re: Capacity Scheduler Issue
Posted by Jian He <jh...@hortonworks.com>.
Did the application get accepted ? If the application gets accepted but
just not able to run, as I said, try increasing
yarn.scheduler.capacity.maximum-am-resource-percent
to a much larger value, maybe 0.8, just for troubleshoot. How large it
should be depends on your requirements.
yarn.scheduler.capacity.maximum-am-resource-percent controls the max
number of concurrently running AMs.
By that I mean the max memory allowed for allocating AM (defined by this
property) divided by per AM memory usage , equals to the max number of
concurrently running AMs.
I have seen both of your queues only allow 1 active application.
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
]
Jian
On Thu, Nov 28, 2013 at 12:23 PM, Adam Kawa <ka...@gmail.com> wrote:
> I see that you have different settings for ACL:
>
> <name>yarn.scheduler.capacity.root.*default*
> .acl_submit_applications</name><value>*yarn,mapred*
> </value></property><property>
> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
> configuredAcls ]
>
> <name>yarn.scheduler.capacity.root.*dev*
> .acl_submit_applications</name><value>*smunnavar,test*</value></property>
>
> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls ]
>
> Do you submit your jobs to dev queue using good user accounts? Did you
> include a right config xml in a previous post? (xml says *smunnavar,test*,
> but RM loads user,test).
>
> As a first troubleshooting steps, could you disable ACL for dev queue to
> check if you can submit a job there?
>
> <property>
> <name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
> <value>*</value>
> </property>
>
>
>
> 2013/11/28 Munna <mu...@gmail.com>
>
>> Hi,
>>
>>
>> I think there is no solution on above issue, so i'll move to fair
>> scheduler.
>>
>> Thanks to all...
>>
>> Munna
>>
>>
>> On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
>>
>>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>>
>>>
>>> what is the best value?
>>>
>>> Tx,
>>> Munna
>>>
>>>
>>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>>>
>>>> The log shows the both queues are properly picked up by the RM.
>>>> If the problem is that your submitted application is not able to run,
>>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>>> this controls the max number of concurrently running AMs in the cluster.
>>>>
>>>> Jian
>>>>
>>>>
>>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>>>
>>>>> Hi Flocks,
>>>>>
>>>>>
>>>>>
>>>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>>>> how I have struggling L….
>>>>>
>>>>>
>>>>>
>>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created
>>>>> 4 Queue’s as per the Capacity Scheduler Documentation and those queues
>>>>> shown in RM UI.
>>>>>
>>>>>
>>>>>
>>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>>> which showing all configured Q’s.
>>>>>
>>>>>
>>>>>
>>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given
>>>>> some idea about that, based on his views I dig more.
>>>>>
>>>>>
>>>>>
>>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>>> “default”, I have tested with *default queue* it works for me. And I
>>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>>> the jobs and its unable to identifying users Queue.
>>>>>
>>>>>
>>>>>
>>>>> I have attached Cap Scheduler configuration file for your information.
>>>>> Some O/P for ur information.
>>>>>
>>>>>
>>>>>
>>>>> *[user@host ~]$ mapred queue -list*
>>>>>
>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>>
>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>>
>>>>> *======================*
>>>>>
>>>>> *Queue Name : dev*
>>>>>
>>>>> *Queue State : running*
>>>>>
>>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>>> CurrentCapacity: 0.0*
>>>>>
>>>>> *======================*
>>>>>
>>>>> *Queue Name : default*
>>>>>
>>>>> *Queue State : running*
>>>>>
>>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>>> CurrentCapacity: 0.0*
>>>>>
>>>>>
>>>>>
>>>>> *RM log Scheduler loading info:*
>>>>>
>>>>> 2013-11-27 08:54:58,521 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>>> ADMINISTER_QUEUE:
>>>>>
>>>>> 2013-11-27 08:54:58,521 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>> Initialized parent-queue root name=root, fullname=root
>>>>>
>>>>> 2013-11-27 08:54:58,534 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>> default*
>>>>>
>>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>>
>>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>>
>>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>>
>>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>
>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>
>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>
>>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>
>>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>>> 100.0f) * userLimitFactor) ]
>>>>>
>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>> ]
>>>>>
>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>
>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>
>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>> absoluteCapacity)]
>>>>>
>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>> clusterResourceMemory]
>>>>>
>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>> configuredMaximumAMResourcePercent ]
>>>>>
>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>
>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>> configuredMaximumAMResourcePercent ]
>>>>>
>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>
>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>
>>>>> state = RUNNING [= configuredState ]
>>>>>
>>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>>> configuredAcls ]
>>>>>
>>>>>
>>>>>
>>>>> 2013-11-27 08:54:58,534 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>>
>>>>> 2013-11-27 08:54:58,543 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>> dev*
>>>>>
>>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>>
>>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>>
>>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>>
>>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>
>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>
>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>
>>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>
>>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>>> 100.0f) * userLimitFactor) ]
>>>>>
>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>> ]
>>>>>
>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>
>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>
>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>> absoluteCapacity)]
>>>>>
>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>> clusterResourceMemory]
>>>>>
>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>> configuredMaximumAMResourcePercent ]
>>>>>
>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>
>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>
>>>>> state = RUNNING [= configuredState ]
>>>>>
>>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>>> configuredAcls ]
>>>>>
>>>>>
>>>>>
>>>>> Can you guys please confirm, did I miss anything on configurations
>>>>> part or is there any bug persist on 2.0.0?
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Munna
>>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>>
>>> --
>>> *Regards*
>>>
>>> *Munna*
>>>
>>
>>
>>
>> --
>> *Regards*
>>
>> *Munna*
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Capacity Scheduler Issue
Posted by Jian He <jh...@hortonworks.com>.
Did the application get accepted ? If the application gets accepted but
just not able to run, as I said, try increasing
yarn.scheduler.capacity.maximum-am-resource-percent
to a much larger value, maybe 0.8, just for troubleshoot. How large it
should be depends on your requirements.
yarn.scheduler.capacity.maximum-am-resource-percent controls the max
number of concurrently running AMs.
By that I mean the max memory allowed for allocating AM (defined by this
property) divided by per AM memory usage , equals to the max number of
concurrently running AMs.
I have seen both of your queues only allow 1 active application.
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
]
Jian
On Thu, Nov 28, 2013 at 12:23 PM, Adam Kawa <ka...@gmail.com> wrote:
> I see that you have different settings for ACL:
>
> <name>yarn.scheduler.capacity.root.*default*
> .acl_submit_applications</name><value>*yarn,mapred*
> </value></property><property>
> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
> configuredAcls ]
>
> <name>yarn.scheduler.capacity.root.*dev*
> .acl_submit_applications</name><value>*smunnavar,test*</value></property>
>
> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls ]
>
> Do you submit your jobs to dev queue using good user accounts? Did you
> include a right config xml in a previous post? (xml says *smunnavar,test*,
> but RM loads user,test).
>
> As a first troubleshooting steps, could you disable ACL for dev queue to
> check if you can submit a job there?
>
> <property>
> <name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
> <value>*</value>
> </property>
>
>
>
> 2013/11/28 Munna <mu...@gmail.com>
>
>> Hi,
>>
>>
>> I think there is no solution on above issue, so i'll move to fair
>> scheduler.
>>
>> Thanks to all...
>>
>> Munna
>>
>>
>> On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
>>
>>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>>
>>>
>>> what is the best value?
>>>
>>> Tx,
>>> Munna
>>>
>>>
>>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>>>
>>>> The log shows the both queues are properly picked up by the RM.
>>>> If the problem is that your submitted application is not able to run,
>>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>>> this controls the max number of concurrently running AMs in the cluster.
>>>>
>>>> Jian
>>>>
>>>>
>>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>>>
>>>>> Hi Flocks,
>>>>>
>>>>>
>>>>>
>>>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>>>> how I have struggling L….
>>>>>
>>>>>
>>>>>
>>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created
>>>>> 4 Queue’s as per the Capacity Scheduler Documentation and those queues
>>>>> shown in RM UI.
>>>>>
>>>>>
>>>>>
>>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>>> which showing all configured Q’s.
>>>>>
>>>>>
>>>>>
>>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given
>>>>> some idea about that, based on his views I dig more.
>>>>>
>>>>>
>>>>>
>>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>>> “default”, I have tested with *default queue* it works for me. And I
>>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>>> the jobs and its unable to identifying users Queue.
>>>>>
>>>>>
>>>>>
>>>>> I have attached Cap Scheduler configuration file for your information.
>>>>> Some O/P for ur information.
>>>>>
>>>>>
>>>>>
>>>>> *[user@host ~]$ mapred queue -list*
>>>>>
>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>>
>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>>
>>>>> *======================*
>>>>>
>>>>> *Queue Name : dev*
>>>>>
>>>>> *Queue State : running*
>>>>>
>>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>>> CurrentCapacity: 0.0*
>>>>>
>>>>> *======================*
>>>>>
>>>>> *Queue Name : default*
>>>>>
>>>>> *Queue State : running*
>>>>>
>>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>>> CurrentCapacity: 0.0*
>>>>>
>>>>>
>>>>>
>>>>> *RM log Scheduler loading info:*
>>>>>
>>>>> 2013-11-27 08:54:58,521 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>>> ADMINISTER_QUEUE:
>>>>>
>>>>> 2013-11-27 08:54:58,521 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>> Initialized parent-queue root name=root, fullname=root
>>>>>
>>>>> 2013-11-27 08:54:58,534 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>> default*
>>>>>
>>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>>
>>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>>
>>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>>
>>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>
>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>
>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>
>>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>
>>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>>> 100.0f) * userLimitFactor) ]
>>>>>
>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>> ]
>>>>>
>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>
>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>
>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>> absoluteCapacity)]
>>>>>
>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>> clusterResourceMemory]
>>>>>
>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>> configuredMaximumAMResourcePercent ]
>>>>>
>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>
>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>> configuredMaximumAMResourcePercent ]
>>>>>
>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>
>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>
>>>>> state = RUNNING [= configuredState ]
>>>>>
>>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>>> configuredAcls ]
>>>>>
>>>>>
>>>>>
>>>>> 2013-11-27 08:54:58,534 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>>
>>>>> 2013-11-27 08:54:58,543 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>> dev*
>>>>>
>>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>>
>>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>>
>>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>>
>>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>
>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>
>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>
>>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>
>>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>>> 100.0f) * userLimitFactor) ]
>>>>>
>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>> ]
>>>>>
>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>
>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>
>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>> absoluteCapacity)]
>>>>>
>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>> clusterResourceMemory]
>>>>>
>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>> configuredMaximumAMResourcePercent ]
>>>>>
>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>
>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>
>>>>> state = RUNNING [= configuredState ]
>>>>>
>>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>>> configuredAcls ]
>>>>>
>>>>>
>>>>>
>>>>> Can you guys please confirm, did I miss anything on configurations
>>>>> part or is there any bug persist on 2.0.0?
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Munna
>>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>>
>>> --
>>> *Regards*
>>>
>>> *Munna*
>>>
>>
>>
>>
>> --
>> *Regards*
>>
>> *Munna*
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Capacity Scheduler Issue
Posted by Jian He <jh...@hortonworks.com>.
Did the application get accepted ? If the application gets accepted but
just not able to run, as I said, try increasing
yarn.scheduler.capacity.maximum-am-resource-percent
to a much larger value, maybe 0.8, just for troubleshoot. How large it
should be depends on your requirements.
yarn.scheduler.capacity.maximum-am-resource-percent controls the max
number of concurrently running AMs.
By that I mean the max memory allowed for allocating AM (defined by this
property) divided by per AM memory usage , equals to the max number of
concurrently running AMs.
I have seen both of your queues only allow 1 active application.
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
]
Jian
On Thu, Nov 28, 2013 at 12:23 PM, Adam Kawa <ka...@gmail.com> wrote:
> I see that you have different settings for ACL:
>
> <name>yarn.scheduler.capacity.root.*default*
> .acl_submit_applications</name><value>*yarn,mapred*
> </value></property><property>
> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
> configuredAcls ]
>
> <name>yarn.scheduler.capacity.root.*dev*
> .acl_submit_applications</name><value>*smunnavar,test*</value></property>
>
> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls ]
>
> Do you submit your jobs to dev queue using good user accounts? Did you
> include a right config xml in a previous post? (xml says *smunnavar,test*,
> but RM loads user,test).
>
> As a first troubleshooting steps, could you disable ACL for dev queue to
> check if you can submit a job there?
>
> <property>
> <name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
> <value>*</value>
> </property>
>
>
>
> 2013/11/28 Munna <mu...@gmail.com>
>
>> Hi,
>>
>>
>> I think there is no solution on above issue, so i'll move to fair
>> scheduler.
>>
>> Thanks to all...
>>
>> Munna
>>
>>
>> On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
>>
>>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>>
>>>
>>> what is the best value?
>>>
>>> Tx,
>>> Munna
>>>
>>>
>>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>>>
>>>> The log shows the both queues are properly picked up by the RM.
>>>> If the problem is that your submitted application is not able to run,
>>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>>> this controls the max number of concurrently running AMs in the cluster.
>>>>
>>>> Jian
>>>>
>>>>
>>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>>>
>>>>> Hi Flocks,
>>>>>
>>>>>
>>>>>
>>>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>>>> how I have struggling L….
>>>>>
>>>>>
>>>>>
>>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created
>>>>> 4 Queue’s as per the Capacity Scheduler Documentation and those queues
>>>>> shown in RM UI.
>>>>>
>>>>>
>>>>>
>>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>>> which showing all configured Q’s.
>>>>>
>>>>>
>>>>>
>>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given
>>>>> some idea about that, based on his views I dig more.
>>>>>
>>>>>
>>>>>
>>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>>> “default”, I have tested with *default queue* it works for me. And I
>>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>>> the jobs and its unable to identifying users Queue.
>>>>>
>>>>>
>>>>>
>>>>> I have attached Cap Scheduler configuration file for your information.
>>>>> Some O/P for ur information.
>>>>>
>>>>>
>>>>>
>>>>> *[user@host ~]$ mapred queue -list*
>>>>>
>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>>
>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>>
>>>>> *======================*
>>>>>
>>>>> *Queue Name : dev*
>>>>>
>>>>> *Queue State : running*
>>>>>
>>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>>> CurrentCapacity: 0.0*
>>>>>
>>>>> *======================*
>>>>>
>>>>> *Queue Name : default*
>>>>>
>>>>> *Queue State : running*
>>>>>
>>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>>> CurrentCapacity: 0.0*
>>>>>
>>>>>
>>>>>
>>>>> *RM log Scheduler loading info:*
>>>>>
>>>>> 2013-11-27 08:54:58,521 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>>> ADMINISTER_QUEUE:
>>>>>
>>>>> 2013-11-27 08:54:58,521 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>> Initialized parent-queue root name=root, fullname=root
>>>>>
>>>>> 2013-11-27 08:54:58,534 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>> default*
>>>>>
>>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>>
>>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>>
>>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>>
>>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>
>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>
>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>
>>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>
>>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>>> 100.0f) * userLimitFactor) ]
>>>>>
>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>> ]
>>>>>
>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>
>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>
>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>> absoluteCapacity)]
>>>>>
>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>> clusterResourceMemory]
>>>>>
>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>> configuredMaximumAMResourcePercent ]
>>>>>
>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>
>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>> configuredMaximumAMResourcePercent ]
>>>>>
>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>
>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>
>>>>> state = RUNNING [= configuredState ]
>>>>>
>>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>>> configuredAcls ]
>>>>>
>>>>>
>>>>>
>>>>> 2013-11-27 08:54:58,534 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>>
>>>>> 2013-11-27 08:54:58,543 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>> dev*
>>>>>
>>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>>
>>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>>
>>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>>
>>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>
>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>
>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>
>>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>
>>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>>> 100.0f) * userLimitFactor) ]
>>>>>
>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>> ]
>>>>>
>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>
>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>
>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>> absoluteCapacity)]
>>>>>
>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>> clusterResourceMemory]
>>>>>
>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>> configuredMaximumAMResourcePercent ]
>>>>>
>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>
>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>
>>>>> state = RUNNING [= configuredState ]
>>>>>
>>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>>> configuredAcls ]
>>>>>
>>>>>
>>>>>
>>>>> Can you guys please confirm, did I miss anything on configurations
>>>>> part or is there any bug persist on 2.0.0?
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Munna
>>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>>
>>> --
>>> *Regards*
>>>
>>> *Munna*
>>>
>>
>>
>>
>> --
>> *Regards*
>>
>> *Munna*
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Capacity Scheduler Issue
Posted by Jian He <jh...@hortonworks.com>.
Did the application get accepted ? If the application gets accepted but
just not able to run, as I said, try increasing
yarn.scheduler.capacity.maximum-am-resource-percent
to a much larger value, maybe 0.8, just for troubleshoot. How large it
should be depends on your requirements.
yarn.scheduler.capacity.maximum-am-resource-percent controls the max
number of concurrently running AMs.
By that I mean the max memory allowed for allocating AM (defined by this
property) divided by per AM memory usage , equals to the max number of
concurrently running AMs.
I have seen both of your queues only allow 1 active application.
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
]
Jian
On Thu, Nov 28, 2013 at 12:23 PM, Adam Kawa <ka...@gmail.com> wrote:
> I see that you have different settings for ACL:
>
> <name>yarn.scheduler.capacity.root.*default*
> .acl_submit_applications</name><value>*yarn,mapred*
> </value></property><property>
> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
> configuredAcls ]
>
> <name>yarn.scheduler.capacity.root.*dev*
> .acl_submit_applications</name><value>*smunnavar,test*</value></property>
>
> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls ]
>
> Do you submit your jobs to dev queue using good user accounts? Did you
> include a right config xml in a previous post? (xml says *smunnavar,test*,
> but RM loads user,test).
>
> As a first troubleshooting steps, could you disable ACL for dev queue to
> check if you can submit a job there?
>
> <property>
> <name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
> <value>*</value>
> </property>
>
>
>
> 2013/11/28 Munna <mu...@gmail.com>
>
>> Hi,
>>
>>
>> I think there is no solution on above issue, so i'll move to fair
>> scheduler.
>>
>> Thanks to all...
>>
>> Munna
>>
>>
>> On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
>>
>>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>>
>>>
>>> what is the best value?
>>>
>>> Tx,
>>> Munna
>>>
>>>
>>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>>>
>>>> The log shows the both queues are properly picked up by the RM.
>>>> If the problem is that your submitted application is not able to run,
>>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>>> this controls the max number of concurrently running AMs in the cluster.
>>>>
>>>> Jian
>>>>
>>>>
>>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>>>
>>>>> Hi Flocks,
>>>>>
>>>>>
>>>>>
>>>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>>>> how I have struggling L….
>>>>>
>>>>>
>>>>>
>>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created
>>>>> 4 Queue’s as per the Capacity Scheduler Documentation and those queues
>>>>> shown in RM UI.
>>>>>
>>>>>
>>>>>
>>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>>> which showing all configured Q’s.
>>>>>
>>>>>
>>>>>
>>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given
>>>>> some idea about that, based on his views I dig more.
>>>>>
>>>>>
>>>>>
>>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>>> “default”, I have tested with *default queue* it works for me. And I
>>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>>> the jobs and its unable to identifying users Queue.
>>>>>
>>>>>
>>>>>
>>>>> I have attached Cap Scheduler configuration file for your information.
>>>>> Some O/P for ur information.
>>>>>
>>>>>
>>>>>
>>>>> *[user@host ~]$ mapred queue -list*
>>>>>
>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>>
>>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>>
>>>>> *======================*
>>>>>
>>>>> *Queue Name : dev*
>>>>>
>>>>> *Queue State : running*
>>>>>
>>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>>> CurrentCapacity: 0.0*
>>>>>
>>>>> *======================*
>>>>>
>>>>> *Queue Name : default*
>>>>>
>>>>> *Queue State : running*
>>>>>
>>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>>> CurrentCapacity: 0.0*
>>>>>
>>>>>
>>>>>
>>>>> *RM log Scheduler loading info:*
>>>>>
>>>>> 2013-11-27 08:54:58,521 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>>> ADMINISTER_QUEUE:
>>>>>
>>>>> 2013-11-27 08:54:58,521 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>>> Initialized parent-queue root name=root, fullname=root
>>>>>
>>>>> 2013-11-27 08:54:58,534 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>> default*
>>>>>
>>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>>
>>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>>
>>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>>
>>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>
>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>
>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>
>>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>
>>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>>> 100.0f) * userLimitFactor) ]
>>>>>
>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>> ]
>>>>>
>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>
>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>
>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>> absoluteCapacity)]
>>>>>
>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>> clusterResourceMemory]
>>>>>
>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>> configuredMaximumAMResourcePercent ]
>>>>>
>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>
>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>> configuredMaximumAMResourcePercent ]
>>>>>
>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>
>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>
>>>>> state = RUNNING [= configuredState ]
>>>>>
>>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>>> configuredAcls ]
>>>>>
>>>>>
>>>>>
>>>>> 2013-11-27 08:54:58,534 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>>
>>>>> 2013-11-27 08:54:58,543 INFO
>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>>> dev*
>>>>>
>>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>>
>>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>>
>>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>>
>>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>>
>>>>> userLimit = 100 [= configuredUserLimit ]
>>>>>
>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>>
>>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>>
>>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>>> 100.0f) * userLimitFactor) ]
>>>>>
>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>>> ]
>>>>>
>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>>
>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>>
>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>>> absoluteCapacity)]
>>>>>
>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>>> clusterResourceMemory]
>>>>>
>>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>>> configuredMaximumAMResourcePercent ]
>>>>>
>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>>
>>>>> numContainers = 0 [= currentNumContainers ]
>>>>>
>>>>> state = RUNNING [= configuredState ]
>>>>>
>>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>>> configuredAcls ]
>>>>>
>>>>>
>>>>>
>>>>> Can you guys please confirm, did I miss anything on configurations
>>>>> part or is there any bug persist on 2.0.0?
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Munna
>>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>>
>>> --
>>> *Regards*
>>>
>>> *Munna*
>>>
>>
>>
>>
>> --
>> *Regards*
>>
>> *Munna*
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Capacity Scheduler Issue
Posted by Adam Kawa <ka...@gmail.com>.
I see that you have different settings for ACL:
<name>yarn.scheduler.capacity.root.*default*
.acl_submit_applications</name><value>*yarn,mapred*
</value></property><property>
acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [= configuredAcls
]
<name>yarn.scheduler.capacity.root.*dev*
.acl_submit_applications</name><value>*smunnavar,test*</value></property>
acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls ]
Do you submit your jobs to dev queue using good user accounts? Did you
include a right config xml in a previous post? (xml says *smunnavar,test*,
but RM loads user,test).
As a first troubleshooting steps, could you disable ACL for dev queue to
check if you can submit a job there?
<property>
<name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
<value>*</value>
</property>
2013/11/28 Munna <mu...@gmail.com>
> Hi,
>
>
> I think there is no solution on above issue, so i'll move to fair
> scheduler.
>
> Thanks to all...
>
> Munna
>
>
> On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
>
>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>
>>
>> what is the best value?
>>
>> Tx,
>> Munna
>>
>>
>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> The log shows the both queues are properly picked up by the RM.
>>> If the problem is that your submitted application is not able to run,
>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>> this controls the max number of concurrently running AMs in the cluster.
>>>
>>> Jian
>>>
>>>
>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>>
>>>> Hi Flocks,
>>>>
>>>>
>>>>
>>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>>> how I have struggling L….
>>>>
>>>>
>>>>
>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created
>>>> 4 Queue’s as per the Capacity Scheduler Documentation and those queues
>>>> shown in RM UI.
>>>>
>>>>
>>>>
>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>> which showing all configured Q’s.
>>>>
>>>>
>>>>
>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>>>> idea about that, based on his views I dig more.
>>>>
>>>>
>>>>
>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>> “default”, I have tested with *default queue* it works for me. And I
>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>> the jobs and its unable to identifying users Queue.
>>>>
>>>>
>>>>
>>>> I have attached Cap Scheduler configuration file for your information.
>>>> Some O/P for ur information.
>>>>
>>>>
>>>>
>>>> *[user@host ~]$ mapred queue -list*
>>>>
>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>
>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>
>>>> *======================*
>>>>
>>>> *Queue Name : dev*
>>>>
>>>> *Queue State : running*
>>>>
>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>> CurrentCapacity: 0.0*
>>>>
>>>> *======================*
>>>>
>>>> *Queue Name : default*
>>>>
>>>> *Queue State : running*
>>>>
>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>> CurrentCapacity: 0.0*
>>>>
>>>>
>>>>
>>>> *RM log Scheduler loading info:*
>>>>
>>>> 2013-11-27 08:54:58,521 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>> ADMINISTER_QUEUE:
>>>>
>>>> 2013-11-27 08:54:58,521 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>> Initialized parent-queue root name=root, fullname=root
>>>>
>>>> 2013-11-27 08:54:58,534 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>> default*
>>>>
>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>
>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>
>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>
>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>
>>>> userLimit = 100 [= configuredUserLimit ]
>>>>
>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>
>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>
>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>> 100.0f) * userLimitFactor) ]
>>>>
>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>> ]
>>>>
>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>
>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>
>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>> absoluteCapacity)]
>>>>
>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>> clusterResourceMemory]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> numContainers = 0 [= currentNumContainers ]
>>>>
>>>> state = RUNNING [= configuredState ]
>>>>
>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>> configuredAcls ]
>>>>
>>>>
>>>>
>>>> 2013-11-27 08:54:58,534 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>
>>>> 2013-11-27 08:54:58,543 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>> dev*
>>>>
>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>
>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>
>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>
>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>
>>>> userLimit = 100 [= configuredUserLimit ]
>>>>
>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>
>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>
>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>> 100.0f) * userLimitFactor) ]
>>>>
>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>> ]
>>>>
>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>
>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>
>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>> absoluteCapacity)]
>>>>
>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>> clusterResourceMemory]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> numContainers = 0 [= currentNumContainers ]
>>>>
>>>> state = RUNNING [= configuredState ]
>>>>
>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>> configuredAcls ]
>>>>
>>>>
>>>>
>>>> Can you guys please confirm, did I miss anything on configurations part
>>>> or is there any bug persist on 2.0.0?
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Munna
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>>
>> --
>> *Regards*
>>
>> *Munna*
>>
>
>
>
> --
> *Regards*
>
> *Munna*
>
Re: Capacity Scheduler Issue
Posted by Adam Kawa <ka...@gmail.com>.
I see that you have different settings for ACL:
<name>yarn.scheduler.capacity.root.*default*
.acl_submit_applications</name><value>*yarn,mapred*
</value></property><property>
acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [= configuredAcls
]
<name>yarn.scheduler.capacity.root.*dev*
.acl_submit_applications</name><value>*smunnavar,test*</value></property>
acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls ]
Do you submit your jobs to dev queue using good user accounts? Did you
include a right config xml in a previous post? (xml says *smunnavar,test*,
but RM loads user,test).
As a first troubleshooting steps, could you disable ACL for dev queue to
check if you can submit a job there?
<property>
<name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
<value>*</value>
</property>
2013/11/28 Munna <mu...@gmail.com>
> Hi,
>
>
> I think there is no solution on above issue, so i'll move to fair
> scheduler.
>
> Thanks to all...
>
> Munna
>
>
> On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
>
>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>
>>
>> what is the best value?
>>
>> Tx,
>> Munna
>>
>>
>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> The log shows the both queues are properly picked up by the RM.
>>> If the problem is that your submitted application is not able to run,
>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>> this controls the max number of concurrently running AMs in the cluster.
>>>
>>> Jian
>>>
>>>
>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>>
>>>> Hi Flocks,
>>>>
>>>>
>>>>
>>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>>> how I have struggling L….
>>>>
>>>>
>>>>
>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created
>>>> 4 Queue’s as per the Capacity Scheduler Documentation and those queues
>>>> shown in RM UI.
>>>>
>>>>
>>>>
>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>> which showing all configured Q’s.
>>>>
>>>>
>>>>
>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>>>> idea about that, based on his views I dig more.
>>>>
>>>>
>>>>
>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>> “default”, I have tested with *default queue* it works for me. And I
>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>> the jobs and its unable to identifying users Queue.
>>>>
>>>>
>>>>
>>>> I have attached Cap Scheduler configuration file for your information.
>>>> Some O/P for ur information.
>>>>
>>>>
>>>>
>>>> *[user@host ~]$ mapred queue -list*
>>>>
>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>
>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>
>>>> *======================*
>>>>
>>>> *Queue Name : dev*
>>>>
>>>> *Queue State : running*
>>>>
>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>> CurrentCapacity: 0.0*
>>>>
>>>> *======================*
>>>>
>>>> *Queue Name : default*
>>>>
>>>> *Queue State : running*
>>>>
>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>> CurrentCapacity: 0.0*
>>>>
>>>>
>>>>
>>>> *RM log Scheduler loading info:*
>>>>
>>>> 2013-11-27 08:54:58,521 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>> ADMINISTER_QUEUE:
>>>>
>>>> 2013-11-27 08:54:58,521 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>> Initialized parent-queue root name=root, fullname=root
>>>>
>>>> 2013-11-27 08:54:58,534 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>> default*
>>>>
>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>
>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>
>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>
>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>
>>>> userLimit = 100 [= configuredUserLimit ]
>>>>
>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>
>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>
>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>> 100.0f) * userLimitFactor) ]
>>>>
>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>> ]
>>>>
>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>
>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>
>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>> absoluteCapacity)]
>>>>
>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>> clusterResourceMemory]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> numContainers = 0 [= currentNumContainers ]
>>>>
>>>> state = RUNNING [= configuredState ]
>>>>
>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>> configuredAcls ]
>>>>
>>>>
>>>>
>>>> 2013-11-27 08:54:58,534 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>
>>>> 2013-11-27 08:54:58,543 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>> dev*
>>>>
>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>
>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>
>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>
>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>
>>>> userLimit = 100 [= configuredUserLimit ]
>>>>
>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>
>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>
>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>> 100.0f) * userLimitFactor) ]
>>>>
>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>> ]
>>>>
>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>
>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>
>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>> absoluteCapacity)]
>>>>
>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>> clusterResourceMemory]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> numContainers = 0 [= currentNumContainers ]
>>>>
>>>> state = RUNNING [= configuredState ]
>>>>
>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>> configuredAcls ]
>>>>
>>>>
>>>>
>>>> Can you guys please confirm, did I miss anything on configurations part
>>>> or is there any bug persist on 2.0.0?
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Munna
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>>
>> --
>> *Regards*
>>
>> *Munna*
>>
>
>
>
> --
> *Regards*
>
> *Munna*
>
Re: Capacity Scheduler Issue
Posted by Adam Kawa <ka...@gmail.com>.
I see that you have different settings for ACL:
<name>yarn.scheduler.capacity.root.*default*
.acl_submit_applications</name><value>*yarn,mapred*
</value></property><property>
acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [= configuredAcls
]
<name>yarn.scheduler.capacity.root.*dev*
.acl_submit_applications</name><value>*smunnavar,test*</value></property>
acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls ]
Do you submit your jobs to dev queue using good user accounts? Did you
include a right config xml in a previous post? (xml says *smunnavar,test*,
but RM loads user,test).
As a first troubleshooting steps, could you disable ACL for dev queue to
check if you can submit a job there?
<property>
<name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
<value>*</value>
</property>
2013/11/28 Munna <mu...@gmail.com>
> Hi,
>
>
> I think there is no solution on above issue, so i'll move to fair
> scheduler.
>
> Thanks to all...
>
> Munna
>
>
> On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
>
>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>
>>
>> what is the best value?
>>
>> Tx,
>> Munna
>>
>>
>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> The log shows the both queues are properly picked up by the RM.
>>> If the problem is that your submitted application is not able to run,
>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>> this controls the max number of concurrently running AMs in the cluster.
>>>
>>> Jian
>>>
>>>
>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>>
>>>> Hi Flocks,
>>>>
>>>>
>>>>
>>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>>> how I have struggling L….
>>>>
>>>>
>>>>
>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created
>>>> 4 Queue’s as per the Capacity Scheduler Documentation and those queues
>>>> shown in RM UI.
>>>>
>>>>
>>>>
>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>> which showing all configured Q’s.
>>>>
>>>>
>>>>
>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>>>> idea about that, based on his views I dig more.
>>>>
>>>>
>>>>
>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>> “default”, I have tested with *default queue* it works for me. And I
>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>> the jobs and its unable to identifying users Queue.
>>>>
>>>>
>>>>
>>>> I have attached Cap Scheduler configuration file for your information.
>>>> Some O/P for ur information.
>>>>
>>>>
>>>>
>>>> *[user@host ~]$ mapred queue -list*
>>>>
>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>
>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>
>>>> *======================*
>>>>
>>>> *Queue Name : dev*
>>>>
>>>> *Queue State : running*
>>>>
>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>> CurrentCapacity: 0.0*
>>>>
>>>> *======================*
>>>>
>>>> *Queue Name : default*
>>>>
>>>> *Queue State : running*
>>>>
>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>> CurrentCapacity: 0.0*
>>>>
>>>>
>>>>
>>>> *RM log Scheduler loading info:*
>>>>
>>>> 2013-11-27 08:54:58,521 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>> ADMINISTER_QUEUE:
>>>>
>>>> 2013-11-27 08:54:58,521 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>> Initialized parent-queue root name=root, fullname=root
>>>>
>>>> 2013-11-27 08:54:58,534 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>> default*
>>>>
>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>
>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>
>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>
>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>
>>>> userLimit = 100 [= configuredUserLimit ]
>>>>
>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>
>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>
>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>> 100.0f) * userLimitFactor) ]
>>>>
>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>> ]
>>>>
>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>
>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>
>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>> absoluteCapacity)]
>>>>
>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>> clusterResourceMemory]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> numContainers = 0 [= currentNumContainers ]
>>>>
>>>> state = RUNNING [= configuredState ]
>>>>
>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>> configuredAcls ]
>>>>
>>>>
>>>>
>>>> 2013-11-27 08:54:58,534 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>
>>>> 2013-11-27 08:54:58,543 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>> dev*
>>>>
>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>
>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>
>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>
>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>
>>>> userLimit = 100 [= configuredUserLimit ]
>>>>
>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>
>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>
>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>> 100.0f) * userLimitFactor) ]
>>>>
>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>> ]
>>>>
>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>
>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>
>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>> absoluteCapacity)]
>>>>
>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>> clusterResourceMemory]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> numContainers = 0 [= currentNumContainers ]
>>>>
>>>> state = RUNNING [= configuredState ]
>>>>
>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>> configuredAcls ]
>>>>
>>>>
>>>>
>>>> Can you guys please confirm, did I miss anything on configurations part
>>>> or is there any bug persist on 2.0.0?
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Munna
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>>
>> --
>> *Regards*
>>
>> *Munna*
>>
>
>
>
> --
> *Regards*
>
> *Munna*
>
Re: Capacity Scheduler Issue
Posted by Adam Kawa <ka...@gmail.com>.
I see that you have different settings for ACL:
<name>yarn.scheduler.capacity.root.*default*
.acl_submit_applications</name><value>*yarn,mapred*
</value></property><property>
acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [= configuredAcls
]
<name>yarn.scheduler.capacity.root.*dev*
.acl_submit_applications</name><value>*smunnavar,test*</value></property>
acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls ]
Do you submit your jobs to dev queue using good user accounts? Did you
include a right config xml in a previous post? (xml says *smunnavar,test*,
but RM loads user,test).
As a first troubleshooting steps, could you disable ACL for dev queue to
check if you can submit a job there?
<property>
<name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
<value>*</value>
</property>
2013/11/28 Munna <mu...@gmail.com>
> Hi,
>
>
> I think there is no solution on above issue, so i'll move to fair
> scheduler.
>
> Thanks to all...
>
> Munna
>
>
> On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
>
>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>
>>
>> what is the best value?
>>
>> Tx,
>> Munna
>>
>>
>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> The log shows the both queues are properly picked up by the RM.
>>> If the problem is that your submitted application is not able to run,
>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>> this controls the max number of concurrently running AMs in the cluster.
>>>
>>> Jian
>>>
>>>
>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>>
>>>> Hi Flocks,
>>>>
>>>>
>>>>
>>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>>> how I have struggling L….
>>>>
>>>>
>>>>
>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created
>>>> 4 Queue’s as per the Capacity Scheduler Documentation and those queues
>>>> shown in RM UI.
>>>>
>>>>
>>>>
>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>> which showing all configured Q’s.
>>>>
>>>>
>>>>
>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>>>> idea about that, based on his views I dig more.
>>>>
>>>>
>>>>
>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>> “default”, I have tested with *default queue* it works for me. And I
>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>> the jobs and its unable to identifying users Queue.
>>>>
>>>>
>>>>
>>>> I have attached Cap Scheduler configuration file for your information.
>>>> Some O/P for ur information.
>>>>
>>>>
>>>>
>>>> *[user@host ~]$ mapred queue -list*
>>>>
>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>
>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>
>>>> *======================*
>>>>
>>>> *Queue Name : dev*
>>>>
>>>> *Queue State : running*
>>>>
>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>> CurrentCapacity: 0.0*
>>>>
>>>> *======================*
>>>>
>>>> *Queue Name : default*
>>>>
>>>> *Queue State : running*
>>>>
>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>> CurrentCapacity: 0.0*
>>>>
>>>>
>>>>
>>>> *RM log Scheduler loading info:*
>>>>
>>>> 2013-11-27 08:54:58,521 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>> ADMINISTER_QUEUE:
>>>>
>>>> 2013-11-27 08:54:58,521 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>> Initialized parent-queue root name=root, fullname=root
>>>>
>>>> 2013-11-27 08:54:58,534 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>> default*
>>>>
>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>
>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>
>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>
>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>
>>>> userLimit = 100 [= configuredUserLimit ]
>>>>
>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>
>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>
>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>> 100.0f) * userLimitFactor) ]
>>>>
>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>> ]
>>>>
>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>
>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>
>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>> absoluteCapacity)]
>>>>
>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>> clusterResourceMemory]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> numContainers = 0 [= currentNumContainers ]
>>>>
>>>> state = RUNNING [= configuredState ]
>>>>
>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>> configuredAcls ]
>>>>
>>>>
>>>>
>>>> 2013-11-27 08:54:58,534 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>
>>>> 2013-11-27 08:54:58,543 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>>> dev*
>>>>
>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>
>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>
>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>
>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>
>>>> userLimit = 100 [= configuredUserLimit ]
>>>>
>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>
>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>
>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>> 100.0f) * userLimitFactor) ]
>>>>
>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>> ]
>>>>
>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>
>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>
>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>> absoluteCapacity)]
>>>>
>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>> clusterResourceMemory]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> numContainers = 0 [= currentNumContainers ]
>>>>
>>>> state = RUNNING [= configuredState ]
>>>>
>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>> configuredAcls ]
>>>>
>>>>
>>>>
>>>> Can you guys please confirm, did I miss anything on configurations part
>>>> or is there any bug persist on 2.0.0?
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Munna
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>>
>> --
>> *Regards*
>>
>> *Munna*
>>
>
>
>
> --
> *Regards*
>
> *Munna*
>
Re: Capacity Scheduler Issue
Posted by Munna <mu...@gmail.com>.
Hi,
I think there is no solution on above issue, so i'll move to fair scheduler.
Thanks to all...
Munna
On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>
>
> what is the best value?
>
> Tx,
> Munna
>
>
> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>
>> The log shows the both queues are properly picked up by the RM.
>> If the problem is that your submitted application is not able to run, you
>> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>> this controls the max number of concurrently running AMs in the cluster.
>>
>> Jian
>>
>>
>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>
>>> Hi Flocks,
>>>
>>>
>>>
>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>> how I have struggling L….
>>>
>>>
>>>
>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
>>> Queue’s as per the Capacity Scheduler Documentation and those queues shown
>>> in RM UI.
>>>
>>>
>>>
>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>> which showing all configured Q’s.
>>>
>>>
>>>
>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>>> idea about that, based on his views I dig more.
>>>
>>>
>>>
>>> After I went to all the RM log, Cap Scheduler initiating only default
>>> “default”, I have tested with *default queue* it works for me. And I
>>> have created one more queue called “dev”, in this Queue User unable to run
>>> the jobs and its unable to identifying users Queue.
>>>
>>>
>>>
>>> I have attached Cap Scheduler configuration file for your information.
>>> Some O/P for ur information.
>>>
>>>
>>>
>>> *[user@host ~]$ mapred queue -list*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>
>>> *======================*
>>>
>>> *Queue Name : dev*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>> CurrentCapacity: 0.0*
>>>
>>> *======================*
>>>
>>> *Queue Name : default*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>> CurrentCapacity: 0.0*
>>>
>>>
>>>
>>> *RM log Scheduler loading info:*
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>> ADMINISTER_QUEUE:
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> Initialized parent-queue root name=root, fullname=root
>>>
>>> 2013-11-27 08:54:58,534 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>> default*
>>>
>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>
>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>
>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>
>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>
>>> userLimit = 100 [= configuredUserLimit ]
>>>
>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>
>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
>>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>
>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>> 100.0f) * userLimitFactor) ]
>>>
>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>> ]
>>>
>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>
>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>
>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>> absoluteCapacity)]
>>>
>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>> clusterResourceMemory]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> numContainers = 0 [= currentNumContainers ]
>>>
>>> state = RUNNING [= configuredState ]
>>>
>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>> configuredAcls ]
>>>
>>>
>>>
>>> 2013-11-27 08:54:58,534 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>
>>> 2013-11-27 08:54:58,543 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>> dev*
>>>
>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>
>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>
>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>
>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>
>>> userLimit = 100 [= configuredUserLimit ]
>>>
>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>
>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
>>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>
>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>> 100.0f) * userLimitFactor) ]
>>>
>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>> ]
>>>
>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>
>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>
>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>> absoluteCapacity)]
>>>
>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>> clusterResourceMemory]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> numContainers = 0 [= currentNumContainers ]
>>>
>>> state = RUNNING [= configuredState ]
>>>
>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>> configuredAcls ]
>>>
>>>
>>>
>>> Can you guys please confirm, did I miss anything on configurations part
>>> or is there any bug persist on 2.0.0?
>>>
>>>
>>>
>>> Thanks
>>>
>>> Munna
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> *Regards*
>
> *Munna*
>
--
*Regards*
*Munna*
Re: Capacity Scheduler Issue
Posted by Munna <mu...@gmail.com>.
Hi,
I think there is no solution on above issue, so i'll move to fair scheduler.
Thanks to all...
Munna
On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>
>
> what is the best value?
>
> Tx,
> Munna
>
>
> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>
>> The log shows the both queues are properly picked up by the RM.
>> If the problem is that your submitted application is not able to run, you
>> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>> this controls the max number of concurrently running AMs in the cluster.
>>
>> Jian
>>
>>
>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>
>>> Hi Flocks,
>>>
>>>
>>>
>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>> how I have struggling L….
>>>
>>>
>>>
>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
>>> Queue’s as per the Capacity Scheduler Documentation and those queues shown
>>> in RM UI.
>>>
>>>
>>>
>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>> which showing all configured Q’s.
>>>
>>>
>>>
>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>>> idea about that, based on his views I dig more.
>>>
>>>
>>>
>>> After I went to all the RM log, Cap Scheduler initiating only default
>>> “default”, I have tested with *default queue* it works for me. And I
>>> have created one more queue called “dev”, in this Queue User unable to run
>>> the jobs and its unable to identifying users Queue.
>>>
>>>
>>>
>>> I have attached Cap Scheduler configuration file for your information.
>>> Some O/P for ur information.
>>>
>>>
>>>
>>> *[user@host ~]$ mapred queue -list*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>
>>> *======================*
>>>
>>> *Queue Name : dev*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>> CurrentCapacity: 0.0*
>>>
>>> *======================*
>>>
>>> *Queue Name : default*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>> CurrentCapacity: 0.0*
>>>
>>>
>>>
>>> *RM log Scheduler loading info:*
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>> ADMINISTER_QUEUE:
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> Initialized parent-queue root name=root, fullname=root
>>>
>>> 2013-11-27 08:54:58,534 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>> default*
>>>
>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>
>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>
>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>
>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>
>>> userLimit = 100 [= configuredUserLimit ]
>>>
>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>
>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
>>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>
>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>> 100.0f) * userLimitFactor) ]
>>>
>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>> ]
>>>
>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>
>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>
>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>> absoluteCapacity)]
>>>
>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>> clusterResourceMemory]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> numContainers = 0 [= currentNumContainers ]
>>>
>>> state = RUNNING [= configuredState ]
>>>
>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>> configuredAcls ]
>>>
>>>
>>>
>>> 2013-11-27 08:54:58,534 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>
>>> 2013-11-27 08:54:58,543 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>> dev*
>>>
>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>
>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>
>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>
>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>
>>> userLimit = 100 [= configuredUserLimit ]
>>>
>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>
>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
>>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>
>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>> 100.0f) * userLimitFactor) ]
>>>
>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>> ]
>>>
>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>
>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>
>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>> absoluteCapacity)]
>>>
>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>> clusterResourceMemory]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> numContainers = 0 [= currentNumContainers ]
>>>
>>> state = RUNNING [= configuredState ]
>>>
>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>> configuredAcls ]
>>>
>>>
>>>
>>> Can you guys please confirm, did I miss anything on configurations part
>>> or is there any bug persist on 2.0.0?
>>>
>>>
>>>
>>> Thanks
>>>
>>> Munna
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> *Regards*
>
> *Munna*
>
--
*Regards*
*Munna*
Re: Capacity Scheduler Issue
Posted by Munna <mu...@gmail.com>.
Hi,
I think there is no solution on above issue, so i'll move to fair scheduler.
Thanks to all...
Munna
On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>
>
> what is the best value?
>
> Tx,
> Munna
>
>
> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>
>> The log shows the both queues are properly picked up by the RM.
>> If the problem is that your submitted application is not able to run, you
>> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>> this controls the max number of concurrently running AMs in the cluster.
>>
>> Jian
>>
>>
>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>
>>> Hi Flocks,
>>>
>>>
>>>
>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>> how I have struggling L….
>>>
>>>
>>>
>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
>>> Queue’s as per the Capacity Scheduler Documentation and those queues shown
>>> in RM UI.
>>>
>>>
>>>
>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>> which showing all configured Q’s.
>>>
>>>
>>>
>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>>> idea about that, based on his views I dig more.
>>>
>>>
>>>
>>> After I went to all the RM log, Cap Scheduler initiating only default
>>> “default”, I have tested with *default queue* it works for me. And I
>>> have created one more queue called “dev”, in this Queue User unable to run
>>> the jobs and its unable to identifying users Queue.
>>>
>>>
>>>
>>> I have attached Cap Scheduler configuration file for your information.
>>> Some O/P for ur information.
>>>
>>>
>>>
>>> *[user@host ~]$ mapred queue -list*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>
>>> *======================*
>>>
>>> *Queue Name : dev*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>> CurrentCapacity: 0.0*
>>>
>>> *======================*
>>>
>>> *Queue Name : default*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>> CurrentCapacity: 0.0*
>>>
>>>
>>>
>>> *RM log Scheduler loading info:*
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>> ADMINISTER_QUEUE:
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> Initialized parent-queue root name=root, fullname=root
>>>
>>> 2013-11-27 08:54:58,534 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>> default*
>>>
>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>
>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>
>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>
>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>
>>> userLimit = 100 [= configuredUserLimit ]
>>>
>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>
>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
>>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>
>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>> 100.0f) * userLimitFactor) ]
>>>
>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>> ]
>>>
>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>
>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>
>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>> absoluteCapacity)]
>>>
>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>> clusterResourceMemory]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> numContainers = 0 [= currentNumContainers ]
>>>
>>> state = RUNNING [= configuredState ]
>>>
>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>> configuredAcls ]
>>>
>>>
>>>
>>> 2013-11-27 08:54:58,534 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>
>>> 2013-11-27 08:54:58,543 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>> dev*
>>>
>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>
>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>
>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>
>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>
>>> userLimit = 100 [= configuredUserLimit ]
>>>
>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>
>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
>>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>
>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>> 100.0f) * userLimitFactor) ]
>>>
>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>> ]
>>>
>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>
>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>
>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>> absoluteCapacity)]
>>>
>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>> clusterResourceMemory]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> numContainers = 0 [= currentNumContainers ]
>>>
>>> state = RUNNING [= configuredState ]
>>>
>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>> configuredAcls ]
>>>
>>>
>>>
>>> Can you guys please confirm, did I miss anything on configurations part
>>> or is there any bug persist on 2.0.0?
>>>
>>>
>>>
>>> Thanks
>>>
>>> Munna
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> *Regards*
>
> *Munna*
>
--
*Regards*
*Munna*
Re: Capacity Scheduler Issue
Posted by Munna <mu...@gmail.com>.
Hi,
I think there is no solution on above issue, so i'll move to fair scheduler.
Thanks to all...
Munna
On Thu, Nov 28, 2013 at 9:11 AM, Munna <mu...@gmail.com> wrote:
> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>
>
> what is the best value?
>
> Tx,
> Munna
>
>
> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
>
>> The log shows the both queues are properly picked up by the RM.
>> If the problem is that your submitted application is not able to run, you
>> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>> this controls the max number of concurrently running AMs in the cluster.
>>
>> Jian
>>
>>
>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>>
>>> Hi Flocks,
>>>
>>>
>>>
>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>> how I have struggling L….
>>>
>>>
>>>
>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
>>> Queue’s as per the Capacity Scheduler Documentation and those queues shown
>>> in RM UI.
>>>
>>>
>>>
>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>> which showing all configured Q’s.
>>>
>>>
>>>
>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>>> idea about that, based on his views I dig more.
>>>
>>>
>>>
>>> After I went to all the RM log, Cap Scheduler initiating only default
>>> “default”, I have tested with *default queue* it works for me. And I
>>> have created one more queue called “dev”, in this Queue User unable to run
>>> the jobs and its unable to identifying users Queue.
>>>
>>>
>>>
>>> I have attached Cap Scheduler configuration file for your information.
>>> Some O/P for ur information.
>>>
>>>
>>>
>>> *[user@host ~]$ mapred queue -list*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>
>>> *======================*
>>>
>>> *Queue Name : dev*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>> CurrentCapacity: 0.0*
>>>
>>> *======================*
>>>
>>> *Queue Name : default*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>> CurrentCapacity: 0.0*
>>>
>>>
>>>
>>> *RM log Scheduler loading info:*
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>> ADMINISTER_QUEUE:
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> Initialized parent-queue root name=root, fullname=root
>>>
>>> 2013-11-27 08:54:58,534 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>> default*
>>>
>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>
>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>
>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>
>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>
>>> userLimit = 100 [= configuredUserLimit ]
>>>
>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>
>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
>>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>
>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>> 100.0f) * userLimitFactor) ]
>>>
>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>> ]
>>>
>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>
>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>
>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>> absoluteCapacity)]
>>>
>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>> clusterResourceMemory]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> numContainers = 0 [= currentNumContainers ]
>>>
>>> state = RUNNING [= configuredState ]
>>>
>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>> configuredAcls ]
>>>
>>>
>>>
>>> 2013-11-27 08:54:58,534 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>
>>> 2013-11-27 08:54:58,543 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>> dev*
>>>
>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>
>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>
>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>
>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>
>>> userLimit = 100 [= configuredUserLimit ]
>>>
>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>
>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
>>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>
>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>> 100.0f) * userLimitFactor) ]
>>>
>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>> ]
>>>
>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>
>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>
>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>> absoluteCapacity)]
>>>
>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>> clusterResourceMemory]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> numContainers = 0 [= currentNumContainers ]
>>>
>>> state = RUNNING [= configuredState ]
>>>
>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>> configuredAcls ]
>>>
>>>
>>>
>>> Can you guys please confirm, did I miss anything on configurations part
>>> or is there any bug persist on 2.0.0?
>>>
>>>
>>>
>>> Thanks
>>>
>>> Munna
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> *Regards*
>
> *Munna*
>
--
*Regards*
*Munna*
Re: Capacity Scheduler Issue
Posted by Munna <mu...@gmail.com>.
I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
what is the best value?
Tx,
Munna
On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
> The log shows the both queues are properly picked up by the RM.
> If the problem is that your submitted application is not able to run, you
> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
> this controls the max number of concurrently running AMs in the cluster.
>
> Jian
>
>
> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>
>> Hi Flocks,
>>
>>
>>
>> Since, last two days I am about to configure Capacity Scheduler. Here,
>> how I have struggling L….
>>
>>
>>
>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
>> Queue’s as per the Capacity Scheduler Documentation and those queues shown
>> in RM UI.
>>
>>
>>
>> After configuration I tried to run Jobs, Cap Scheduler not identified
>> that queue’s. where I have check queues list with “mapred queue –list”,
>> which showing all configured Q’s.
>>
>>
>>
>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>> idea about that, based on his views I dig more.
>>
>>
>>
>> After I went to all the RM log, Cap Scheduler initiating only default
>> “default”, I have tested with *default queue* it works for me. And I
>> have created one more queue called “dev”, in this Queue User unable to run
>> the jobs and its unable to identifying users Queue.
>>
>>
>>
>> I have attached Cap Scheduler configuration file for your information.
>> Some O/P for ur information.
>>
>>
>>
>> *[user@host ~]$ mapred queue -list*
>>
>> *13/11/27 09:26:38 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>
>> *13/11/27 09:26:38 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>
>> *======================*
>>
>> *Queue Name : dev*
>>
>> *Queue State : running*
>>
>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>> CurrentCapacity: 0.0*
>>
>> *======================*
>>
>> *Queue Name : default*
>>
>> *Queue State : running*
>>
>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
>> 0.0*
>>
>>
>>
>> *RM log Scheduler loading info:*
>>
>> 2013-11-27 08:54:58,521 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>> ADMINISTER_QUEUE:
>>
>> 2013-11-27 08:54:58,521 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>> Initialized parent-queue root name=root, fullname=root
>>
>> 2013-11-27 08:54:58,534 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>> default*
>>
>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>
>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>
>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>
>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>
>> userLimit = 100 [= configuredUserLimit ]
>>
>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>
>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>
>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>> 100.0f) * userLimitFactor) ]
>>
>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>> ]
>>
>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>
>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>
>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>> absoluteCapacity)]
>>
>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> numContainers = 0 [= currentNumContainers ]
>>
>> state = RUNNING [= configuredState ]
>>
>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>> configuredAcls ]
>>
>>
>>
>> 2013-11-27 08:54:58,534 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>
>> 2013-11-27 08:54:58,543 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>> dev*
>>
>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>
>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>
>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>
>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>
>> userLimit = 100 [= configuredUserLimit ]
>>
>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>
>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>
>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>> 100.0f) * userLimitFactor) ]
>>
>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>> ]
>>
>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>
>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>
>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>> absoluteCapacity)]
>>
>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> numContainers = 0 [= currentNumContainers ]
>>
>> state = RUNNING [= configuredState ]
>>
>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>> configuredAcls ]
>>
>>
>>
>> Can you guys please confirm, did I miss anything on configurations part
>> or is there any bug persist on 2.0.0?
>>
>>
>>
>> Thanks
>>
>> Munna
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
*Regards*
*Munna*
Re: Capacity Scheduler Issue
Posted by Munna <mu...@gmail.com>.
I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
what is the best value?
Tx,
Munna
On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
> The log shows the both queues are properly picked up by the RM.
> If the problem is that your submitted application is not able to run, you
> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
> this controls the max number of concurrently running AMs in the cluster.
>
> Jian
>
>
> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>
>> Hi Flocks,
>>
>>
>>
>> Since, last two days I am about to configure Capacity Scheduler. Here,
>> how I have struggling L….
>>
>>
>>
>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
>> Queue’s as per the Capacity Scheduler Documentation and those queues shown
>> in RM UI.
>>
>>
>>
>> After configuration I tried to run Jobs, Cap Scheduler not identified
>> that queue’s. where I have check queues list with “mapred queue –list”,
>> which showing all configured Q’s.
>>
>>
>>
>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>> idea about that, based on his views I dig more.
>>
>>
>>
>> After I went to all the RM log, Cap Scheduler initiating only default
>> “default”, I have tested with *default queue* it works for me. And I
>> have created one more queue called “dev”, in this Queue User unable to run
>> the jobs and its unable to identifying users Queue.
>>
>>
>>
>> I have attached Cap Scheduler configuration file for your information.
>> Some O/P for ur information.
>>
>>
>>
>> *[user@host ~]$ mapred queue -list*
>>
>> *13/11/27 09:26:38 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>
>> *13/11/27 09:26:38 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>
>> *======================*
>>
>> *Queue Name : dev*
>>
>> *Queue State : running*
>>
>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>> CurrentCapacity: 0.0*
>>
>> *======================*
>>
>> *Queue Name : default*
>>
>> *Queue State : running*
>>
>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
>> 0.0*
>>
>>
>>
>> *RM log Scheduler loading info:*
>>
>> 2013-11-27 08:54:58,521 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>> ADMINISTER_QUEUE:
>>
>> 2013-11-27 08:54:58,521 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>> Initialized parent-queue root name=root, fullname=root
>>
>> 2013-11-27 08:54:58,534 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>> default*
>>
>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>
>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>
>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>
>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>
>> userLimit = 100 [= configuredUserLimit ]
>>
>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>
>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>
>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>> 100.0f) * userLimitFactor) ]
>>
>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>> ]
>>
>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>
>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>
>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>> absoluteCapacity)]
>>
>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> numContainers = 0 [= currentNumContainers ]
>>
>> state = RUNNING [= configuredState ]
>>
>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>> configuredAcls ]
>>
>>
>>
>> 2013-11-27 08:54:58,534 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>
>> 2013-11-27 08:54:58,543 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>> dev*
>>
>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>
>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>
>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>
>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>
>> userLimit = 100 [= configuredUserLimit ]
>>
>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>
>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>
>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>> 100.0f) * userLimitFactor) ]
>>
>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>> ]
>>
>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>
>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>
>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>> absoluteCapacity)]
>>
>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> numContainers = 0 [= currentNumContainers ]
>>
>> state = RUNNING [= configuredState ]
>>
>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>> configuredAcls ]
>>
>>
>>
>> Can you guys please confirm, did I miss anything on configurations part
>> or is there any bug persist on 2.0.0?
>>
>>
>>
>> Thanks
>>
>> Munna
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
*Regards*
*Munna*
Re: Capacity Scheduler Issue
Posted by Munna <mu...@gmail.com>.
I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
what is the best value?
Tx,
Munna
On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
> The log shows the both queues are properly picked up by the RM.
> If the problem is that your submitted application is not able to run, you
> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
> this controls the max number of concurrently running AMs in the cluster.
>
> Jian
>
>
> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>
>> Hi Flocks,
>>
>>
>>
>> Since, last two days I am about to configure Capacity Scheduler. Here,
>> how I have struggling L….
>>
>>
>>
>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
>> Queue’s as per the Capacity Scheduler Documentation and those queues shown
>> in RM UI.
>>
>>
>>
>> After configuration I tried to run Jobs, Cap Scheduler not identified
>> that queue’s. where I have check queues list with “mapred queue –list”,
>> which showing all configured Q’s.
>>
>>
>>
>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>> idea about that, based on his views I dig more.
>>
>>
>>
>> After I went to all the RM log, Cap Scheduler initiating only default
>> “default”, I have tested with *default queue* it works for me. And I
>> have created one more queue called “dev”, in this Queue User unable to run
>> the jobs and its unable to identifying users Queue.
>>
>>
>>
>> I have attached Cap Scheduler configuration file for your information.
>> Some O/P for ur information.
>>
>>
>>
>> *[user@host ~]$ mapred queue -list*
>>
>> *13/11/27 09:26:38 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>
>> *13/11/27 09:26:38 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>
>> *======================*
>>
>> *Queue Name : dev*
>>
>> *Queue State : running*
>>
>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>> CurrentCapacity: 0.0*
>>
>> *======================*
>>
>> *Queue Name : default*
>>
>> *Queue State : running*
>>
>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
>> 0.0*
>>
>>
>>
>> *RM log Scheduler loading info:*
>>
>> 2013-11-27 08:54:58,521 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>> ADMINISTER_QUEUE:
>>
>> 2013-11-27 08:54:58,521 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>> Initialized parent-queue root name=root, fullname=root
>>
>> 2013-11-27 08:54:58,534 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>> default*
>>
>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>
>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>
>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>
>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>
>> userLimit = 100 [= configuredUserLimit ]
>>
>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>
>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>
>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>> 100.0f) * userLimitFactor) ]
>>
>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>> ]
>>
>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>
>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>
>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>> absoluteCapacity)]
>>
>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> numContainers = 0 [= currentNumContainers ]
>>
>> state = RUNNING [= configuredState ]
>>
>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>> configuredAcls ]
>>
>>
>>
>> 2013-11-27 08:54:58,534 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>
>> 2013-11-27 08:54:58,543 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>> dev*
>>
>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>
>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>
>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>
>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>
>> userLimit = 100 [= configuredUserLimit ]
>>
>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>
>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>
>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>> 100.0f) * userLimitFactor) ]
>>
>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>> ]
>>
>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>
>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>
>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>> absoluteCapacity)]
>>
>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> numContainers = 0 [= currentNumContainers ]
>>
>> state = RUNNING [= configuredState ]
>>
>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>> configuredAcls ]
>>
>>
>>
>> Can you guys please confirm, did I miss anything on configurations part
>> or is there any bug persist on 2.0.0?
>>
>>
>>
>> Thanks
>>
>> Munna
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
*Regards*
*Munna*
Re: Capacity Scheduler Issue
Posted by Munna <mu...@gmail.com>.
I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
what is the best value?
Tx,
Munna
On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jh...@hortonworks.com> wrote:
> The log shows the both queues are properly picked up by the RM.
> If the problem is that your submitted application is not able to run, you
> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
> this controls the max number of concurrently running AMs in the cluster.
>
> Jian
>
>
> On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
>
>> Hi Flocks,
>>
>>
>>
>> Since, last two days I am about to configure Capacity Scheduler. Here,
>> how I have struggling L….
>>
>>
>>
>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
>> Queue’s as per the Capacity Scheduler Documentation and those queues shown
>> in RM UI.
>>
>>
>>
>> After configuration I tried to run Jobs, Cap Scheduler not identified
>> that queue’s. where I have check queues list with “mapred queue –list”,
>> which showing all configured Q’s.
>>
>>
>>
>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>> idea about that, based on his views I dig more.
>>
>>
>>
>> After I went to all the RM log, Cap Scheduler initiating only default
>> “default”, I have tested with *default queue* it works for me. And I
>> have created one more queue called “dev”, in this Queue User unable to run
>> the jobs and its unable to identifying users Queue.
>>
>>
>>
>> I have attached Cap Scheduler configuration file for your information.
>> Some O/P for ur information.
>>
>>
>>
>> *[user@host ~]$ mapred queue -list*
>>
>> *13/11/27 09:26:38 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>
>> *13/11/27 09:26:38 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>
>> *======================*
>>
>> *Queue Name : dev*
>>
>> *Queue State : running*
>>
>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>> CurrentCapacity: 0.0*
>>
>> *======================*
>>
>> *Queue Name : default*
>>
>> *Queue State : running*
>>
>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
>> 0.0*
>>
>>
>>
>> *RM log Scheduler loading info:*
>>
>> 2013-11-27 08:54:58,521 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>> ADMINISTER_QUEUE:
>>
>> 2013-11-27 08:54:58,521 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>> Initialized parent-queue root name=root, fullname=root
>>
>> 2013-11-27 08:54:58,534 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>> default*
>>
>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>
>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>
>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>
>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>
>> userLimit = 100 [= configuredUserLimit ]
>>
>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>
>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>
>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>> 100.0f) * userLimitFactor) ]
>>
>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>> ]
>>
>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>
>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>
>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>> absoluteCapacity)]
>>
>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> numContainers = 0 [= currentNumContainers ]
>>
>> state = RUNNING [= configuredState ]
>>
>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>> configuredAcls ]
>>
>>
>>
>> 2013-11-27 08:54:58,534 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>
>> 2013-11-27 08:54:58,543 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>> dev*
>>
>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>
>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>
>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>
>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>
>> userLimit = 100 [= configuredUserLimit ]
>>
>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>
>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>
>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>> 100.0f) * userLimitFactor) ]
>>
>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>> ]
>>
>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>
>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>
>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>> absoluteCapacity)]
>>
>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> numContainers = 0 [= currentNumContainers ]
>>
>> state = RUNNING [= configuredState ]
>>
>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>> configuredAcls ]
>>
>>
>>
>> Can you guys please confirm, did I miss anything on configurations part
>> or is there any bug persist on 2.0.0?
>>
>>
>>
>> Thanks
>>
>> Munna
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
*Regards*
*Munna*
Re: Capacity Scheduler Issue
Posted by Jian He <jh...@hortonworks.com>.
The log shows the both queues are properly picked up by the RM.
If the problem is that your submitted application is not able to run, you
may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
this controls the max number of concurrently running AMs in the cluster.
Jian
On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
> Hi Flocks,
>
>
>
> Since, last two days I am about to configure Capacity Scheduler. Here, how
> I have struggling L….
>
>
>
> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
> Queue’s as per the Capacity Scheduler Documentation and those queues shown
> in RM UI.
>
>
>
> After configuration I tried to run Jobs, Cap Scheduler not identified that
> queue’s. where I have check queues list with “mapred queue –list”, which
> showing all configured Q’s.
>
>
>
> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
> idea about that, based on his views I dig more.
>
>
>
> After I went to all the RM log, Cap Scheduler initiating only default
> “default”, I have tested with *default queue* it works for me. And I have
> created one more queue called “dev”, in this Queue User unable to run the
> jobs and its unable to identifying users Queue.
>
>
>
> I have attached Cap Scheduler configuration file for your information.
> Some O/P for ur information.
>
>
>
> *[user@host ~]$ mapred queue -list*
>
> *13/11/27 09:26:38 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>
> *13/11/27 09:26:38 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>
> *======================*
>
> *Queue Name : dev*
>
> *Queue State : running*
>
> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
> CurrentCapacity: 0.0*
>
> *======================*
>
> *Queue Name : default*
>
> *Queue State : running*
>
> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
> 0.0*
>
>
>
> *RM log Scheduler loading info:*
>
> 2013-11-27 08:54:58,521 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
> ADMINISTER_QUEUE:
>
> 2013-11-27 08:54:58,521 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Initialized parent-queue root name=root, fullname=root
>
> 2013-11-27 08:54:58,534 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
> default*
>
> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>
> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>
> maxCapacity = 1.0 [= configuredMaxCapacity ]
>
> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>
> userLimit = 100 [= configuredUserLimit ]
>
> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>
> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>
> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
> 100.0f) * userLimitFactor) ]
>
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>
> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
> (userLimit / 100.0f) * userLimitFactor),1) ]
>
> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
> absoluteCapacity)]
>
> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> numContainers = 0 [= currentNumContainers ]
>
> state = RUNNING [= configuredState ]
>
> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
> configuredAcls ]
>
>
>
> 2013-11-27 08:54:58,534 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>
> 2013-11-27 08:54:58,543 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
> dev*
>
> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>
> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>
> maxCapacity = 0.5 [= configuredMaxCapacity ]
>
> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>
> userLimit = 100 [= configuredUserLimit ]
>
> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>
> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>
> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
> 100.0f) * userLimitFactor) ]
>
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>
> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
> (userLimit / 100.0f) * userLimitFactor),1) ]
>
> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
> absoluteCapacity)]
>
> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> numContainers = 0 [= currentNumContainers ]
>
> state = RUNNING [= configuredState ]
>
> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls
> ]
>
>
>
> Can you guys please confirm, did I miss anything on configurations part or
> is there any bug persist on 2.0.0?
>
>
>
> Thanks
>
> Munna
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Capacity Scheduler Issue
Posted by Jian He <jh...@hortonworks.com>.
The log shows the both queues are properly picked up by the RM.
If the problem is that your submitted application is not able to run, you
may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
this controls the max number of concurrently running AMs in the cluster.
Jian
On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
> Hi Flocks,
>
>
>
> Since, last two days I am about to configure Capacity Scheduler. Here, how
> I have struggling L….
>
>
>
> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
> Queue’s as per the Capacity Scheduler Documentation and those queues shown
> in RM UI.
>
>
>
> After configuration I tried to run Jobs, Cap Scheduler not identified that
> queue’s. where I have check queues list with “mapred queue –list”, which
> showing all configured Q’s.
>
>
>
> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
> idea about that, based on his views I dig more.
>
>
>
> After I went to all the RM log, Cap Scheduler initiating only default
> “default”, I have tested with *default queue* it works for me. And I have
> created one more queue called “dev”, in this Queue User unable to run the
> jobs and its unable to identifying users Queue.
>
>
>
> I have attached Cap Scheduler configuration file for your information.
> Some O/P for ur information.
>
>
>
> *[user@host ~]$ mapred queue -list*
>
> *13/11/27 09:26:38 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>
> *13/11/27 09:26:38 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>
> *======================*
>
> *Queue Name : dev*
>
> *Queue State : running*
>
> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
> CurrentCapacity: 0.0*
>
> *======================*
>
> *Queue Name : default*
>
> *Queue State : running*
>
> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
> 0.0*
>
>
>
> *RM log Scheduler loading info:*
>
> 2013-11-27 08:54:58,521 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
> ADMINISTER_QUEUE:
>
> 2013-11-27 08:54:58,521 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Initialized parent-queue root name=root, fullname=root
>
> 2013-11-27 08:54:58,534 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
> default*
>
> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>
> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>
> maxCapacity = 1.0 [= configuredMaxCapacity ]
>
> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>
> userLimit = 100 [= configuredUserLimit ]
>
> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>
> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>
> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
> 100.0f) * userLimitFactor) ]
>
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>
> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
> (userLimit / 100.0f) * userLimitFactor),1) ]
>
> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
> absoluteCapacity)]
>
> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> numContainers = 0 [= currentNumContainers ]
>
> state = RUNNING [= configuredState ]
>
> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
> configuredAcls ]
>
>
>
> 2013-11-27 08:54:58,534 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>
> 2013-11-27 08:54:58,543 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
> dev*
>
> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>
> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>
> maxCapacity = 0.5 [= configuredMaxCapacity ]
>
> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>
> userLimit = 100 [= configuredUserLimit ]
>
> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>
> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>
> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
> 100.0f) * userLimitFactor) ]
>
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>
> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
> (userLimit / 100.0f) * userLimitFactor),1) ]
>
> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
> absoluteCapacity)]
>
> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> numContainers = 0 [= currentNumContainers ]
>
> state = RUNNING [= configuredState ]
>
> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls
> ]
>
>
>
> Can you guys please confirm, did I miss anything on configurations part or
> is there any bug persist on 2.0.0?
>
>
>
> Thanks
>
> Munna
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Capacity Scheduler Issue
Posted by Jian He <jh...@hortonworks.com>.
The log shows the both queues are properly picked up by the RM.
If the problem is that your submitted application is not able to run, you
may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
this controls the max number of concurrently running AMs in the cluster.
Jian
On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
> Hi Flocks,
>
>
>
> Since, last two days I am about to configure Capacity Scheduler. Here, how
> I have struggling L….
>
>
>
> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
> Queue’s as per the Capacity Scheduler Documentation and those queues shown
> in RM UI.
>
>
>
> After configuration I tried to run Jobs, Cap Scheduler not identified that
> queue’s. where I have check queues list with “mapred queue –list”, which
> showing all configured Q’s.
>
>
>
> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
> idea about that, based on his views I dig more.
>
>
>
> After I went to all the RM log, Cap Scheduler initiating only default
> “default”, I have tested with *default queue* it works for me. And I have
> created one more queue called “dev”, in this Queue User unable to run the
> jobs and its unable to identifying users Queue.
>
>
>
> I have attached Cap Scheduler configuration file for your information.
> Some O/P for ur information.
>
>
>
> *[user@host ~]$ mapred queue -list*
>
> *13/11/27 09:26:38 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>
> *13/11/27 09:26:38 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>
> *======================*
>
> *Queue Name : dev*
>
> *Queue State : running*
>
> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
> CurrentCapacity: 0.0*
>
> *======================*
>
> *Queue Name : default*
>
> *Queue State : running*
>
> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
> 0.0*
>
>
>
> *RM log Scheduler loading info:*
>
> 2013-11-27 08:54:58,521 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
> ADMINISTER_QUEUE:
>
> 2013-11-27 08:54:58,521 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Initialized parent-queue root name=root, fullname=root
>
> 2013-11-27 08:54:58,534 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
> default*
>
> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>
> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>
> maxCapacity = 1.0 [= configuredMaxCapacity ]
>
> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>
> userLimit = 100 [= configuredUserLimit ]
>
> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>
> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>
> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
> 100.0f) * userLimitFactor) ]
>
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>
> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
> (userLimit / 100.0f) * userLimitFactor),1) ]
>
> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
> absoluteCapacity)]
>
> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> numContainers = 0 [= currentNumContainers ]
>
> state = RUNNING [= configuredState ]
>
> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
> configuredAcls ]
>
>
>
> 2013-11-27 08:54:58,534 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>
> 2013-11-27 08:54:58,543 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
> dev*
>
> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>
> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>
> maxCapacity = 0.5 [= configuredMaxCapacity ]
>
> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>
> userLimit = 100 [= configuredUserLimit ]
>
> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>
> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>
> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
> 100.0f) * userLimitFactor) ]
>
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>
> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
> (userLimit / 100.0f) * userLimitFactor),1) ]
>
> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
> absoluteCapacity)]
>
> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> numContainers = 0 [= currentNumContainers ]
>
> state = RUNNING [= configuredState ]
>
> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls
> ]
>
>
>
> Can you guys please confirm, did I miss anything on configurations part or
> is there any bug persist on 2.0.0?
>
>
>
> Thanks
>
> Munna
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Capacity Scheduler Issue
Posted by Jian He <jh...@hortonworks.com>.
The log shows the both queues are properly picked up by the RM.
If the problem is that your submitted application is not able to run, you
may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
this controls the max number of concurrently running AMs in the cluster.
Jian
On Wed, Nov 27, 2013 at 9:42 AM, Munna <mu...@gmail.com> wrote:
> Hi Flocks,
>
>
>
> Since, last two days I am about to configure Capacity Scheduler. Here, how
> I have struggling L….
>
>
>
> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
> Queue’s as per the Capacity Scheduler Documentation and those queues shown
> in RM UI.
>
>
>
> After configuration I tried to run Jobs, Cap Scheduler not identified that
> queue’s. where I have check queues list with “mapred queue –list”, which
> showing all configured Q’s.
>
>
>
> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
> idea about that, based on his views I dig more.
>
>
>
> After I went to all the RM log, Cap Scheduler initiating only default
> “default”, I have tested with *default queue* it works for me. And I have
> created one more queue called “dev”, in this Queue User unable to run the
> jobs and its unable to identifying users Queue.
>
>
>
> I have attached Cap Scheduler configuration file for your information.
> Some O/P for ur information.
>
>
>
> *[user@host ~]$ mapred queue -list*
>
> *13/11/27 09:26:38 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>
> *13/11/27 09:26:38 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>
> *======================*
>
> *Queue Name : dev*
>
> *Queue State : running*
>
> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
> CurrentCapacity: 0.0*
>
> *======================*
>
> *Queue Name : default*
>
> *Queue State : running*
>
> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
> 0.0*
>
>
>
> *RM log Scheduler loading info:*
>
> 2013-11-27 08:54:58,521 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
> ADMINISTER_QUEUE:
>
> 2013-11-27 08:54:58,521 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Initialized parent-queue root name=root, fullname=root
>
> 2013-11-27 08:54:58,534 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
> default*
>
> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>
> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>
> maxCapacity = 1.0 [= configuredMaxCapacity ]
>
> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>
> userLimit = 100 [= configuredUserLimit ]
>
> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>
> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>
> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
> 100.0f) * userLimitFactor) ]
>
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>
> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
> (userLimit / 100.0f) * userLimitFactor),1) ]
>
> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
> absoluteCapacity)]
>
> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> numContainers = 0 [= currentNumContainers ]
>
> state = RUNNING [= configuredState ]
>
> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
> configuredAcls ]
>
>
>
> 2013-11-27 08:54:58,534 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>
> 2013-11-27 08:54:58,543 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
> dev*
>
> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>
> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>
> maxCapacity = 0.5 [= configuredMaxCapacity ]
>
> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>
> userLimit = 100 [= configuredUserLimit ]
>
> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>
> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>
> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
> 100.0f) * userLimitFactor) ]
>
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>
> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
> (userLimit / 100.0f) * userLimitFactor),1) ]
>
> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
> absoluteCapacity)]
>
> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> numContainers = 0 [= currentNumContainers ]
>
> state = RUNNING [= configuredState ]
>
> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls
> ]
>
>
>
> Can you guys please confirm, did I miss anything on configurations part or
> is there any bug persist on 2.0.0?
>
>
>
> Thanks
>
> Munna
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.