You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ch huang <ju...@gmail.com> on 2013/11/26 10:20:06 UTC
default capacity scheduler only one job in running status
hi,maillist:
i set the following option in yarn-site.xml ,let yarn framework
to use capacity scheduler,but i submit three job,only one job in running
status,other two stay in accepted status,why ,the default queue only 50%
capacity used,i do not know why?
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
yes yarn.scheduler.capacity.maximum-am-resource-percent option is a limit,
i increase the value from 0.1 to 0.8 but ,it can still allow 2 job run
parallel,if i laugh
third job ,it still need wait,why? the whole cluster only can run 2 job
parallel?
On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <de...@apache.org> wrote:
> Could you check the below configuration in capacity-scheduler.xml,
> whether is it causing to run only one AM.
>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
>
>
> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i set the following option in yarn-site.xml ,let yarn
>> framework to use capacity scheduler,but i submit three job,only one job in
>> running status,other two stay in accepted status,why ,the default queue
>> only 50% capacity used,i do not know why?
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>> </property>
>>
>
>
>
> --
>
>
> Thanks
> Devaraj K
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
# yarn application -list
13/11/27 09:13:02 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/11/27 09:13:03 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
Total Applications:2
Application-Id Application-Name
User Queue State
Final-State Tracking-URL
application_1385514714001_0002 select count(1) from
test_carl_pageview_p(Stage-1) hive
default ACCEPTED
UNDEFINED N/A
application_1385514714001_0001 select count(1) from
test_carl_pageview_p(Stage-1) hive
default RUNNING
UNDEFINED CHBM222:59768
13/11/27 09:13:03 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
here is my capacity-scheduler.xml content
<configuration>
<property>
<name>yarn.scheduler.capacity.maximum-applications</name>
<value>10000</value>
<description>
Maximum number of applications that can be pending and running.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.3</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator</value>
<description>
The ResourceCalculator implementation to be used to compare
Resources in the scheduler.
The default i.e. DefaultResourceCalculator only uses Memory while
DominantResourceCalculator uses dominant-resource to compare
multi-dimensional resources such as Memory, CPU etc.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>default</value>
<description>
The queues at the this level (root is the root queue).
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.capacity</name>
<value>100</value>
<description>Default queue target capacity.</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
<value>0.3</value>
<description>
Default queue user limit a percentage from 0.0 to 1.0.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.minimum-user-limit-percent</name>
<value>25</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
<value>100</value>
<description>
The maximum capacity of the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.state</name>
<value>RUNNING</value>
<description>
The state of the default queue. State can be one of RUNNING or
STOPPED.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
<value>*</value>
<description>
The ACL of who can submit jobs to the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
<value>*</value>
<description>
The ACL of who can administer jobs on the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.node-locality-delay</name>
<value>-1</value>
<description>
Number of missed scheduling opportunities after which the
CapacityScheduler
attempts to schedule rack-local containers.
Typically this should be set to number of racks in the cluster, this
feature is disabled by default, set to -1.
</description>
</property>
</configuration>
On Wed, Nov 27, 2013 at 8:33 AM, ch huang <ju...@gmail.com> wrote:
> my config is
>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
>
>
> On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <de...@apache.org> wrote:
>
>> Could you check the below configuration in capacity-scheduler.xml,
>> whether is it causing to run only one AM.
>>
>> <property>
>> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
>> <value>0.1</value>
>> <description>
>> Maximum percent of resources in the cluster which can be used to
>> run
>> application masters i.e. controls number of concurrent running
>> applications.
>> </description>
>> </property>
>>
>>
>> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
>>
>>> hi,maillist:
>>> i set the following option in yarn-site.xml ,let yarn
>>> framework to use capacity scheduler,but i submit three job,only one job in
>>> running status,other two stay in accepted status,why ,the default queue
>>> only 50% capacity used,i do not know why?
>>>
>>> <property>
>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>
>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>> </property>
>>>
>>
>>
>>
>> --
>>
>>
>> Thanks
>> Devaraj K
>>
>
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
# yarn application -list
13/11/27 09:13:02 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/11/27 09:13:03 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
Total Applications:2
Application-Id Application-Name
User Queue State
Final-State Tracking-URL
application_1385514714001_0002 select count(1) from
test_carl_pageview_p(Stage-1) hive
default ACCEPTED
UNDEFINED N/A
application_1385514714001_0001 select count(1) from
test_carl_pageview_p(Stage-1) hive
default RUNNING
UNDEFINED CHBM222:59768
13/11/27 09:13:03 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
here is my capacity-scheduler.xml content
<configuration>
<property>
<name>yarn.scheduler.capacity.maximum-applications</name>
<value>10000</value>
<description>
Maximum number of applications that can be pending and running.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.3</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator</value>
<description>
The ResourceCalculator implementation to be used to compare
Resources in the scheduler.
The default i.e. DefaultResourceCalculator only uses Memory while
DominantResourceCalculator uses dominant-resource to compare
multi-dimensional resources such as Memory, CPU etc.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>default</value>
<description>
The queues at the this level (root is the root queue).
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.capacity</name>
<value>100</value>
<description>Default queue target capacity.</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
<value>0.3</value>
<description>
Default queue user limit a percentage from 0.0 to 1.0.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.minimum-user-limit-percent</name>
<value>25</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
<value>100</value>
<description>
The maximum capacity of the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.state</name>
<value>RUNNING</value>
<description>
The state of the default queue. State can be one of RUNNING or
STOPPED.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
<value>*</value>
<description>
The ACL of who can submit jobs to the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
<value>*</value>
<description>
The ACL of who can administer jobs on the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.node-locality-delay</name>
<value>-1</value>
<description>
Number of missed scheduling opportunities after which the
CapacityScheduler
attempts to schedule rack-local containers.
Typically this should be set to number of racks in the cluster, this
feature is disabled by default, set to -1.
</description>
</property>
</configuration>
On Wed, Nov 27, 2013 at 8:33 AM, ch huang <ju...@gmail.com> wrote:
> my config is
>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
>
>
> On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <de...@apache.org> wrote:
>
>> Could you check the below configuration in capacity-scheduler.xml,
>> whether is it causing to run only one AM.
>>
>> <property>
>> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
>> <value>0.1</value>
>> <description>
>> Maximum percent of resources in the cluster which can be used to
>> run
>> application masters i.e. controls number of concurrent running
>> applications.
>> </description>
>> </property>
>>
>>
>> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
>>
>>> hi,maillist:
>>> i set the following option in yarn-site.xml ,let yarn
>>> framework to use capacity scheduler,but i submit three job,only one job in
>>> running status,other two stay in accepted status,why ,the default queue
>>> only 50% capacity used,i do not know why?
>>>
>>> <property>
>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>
>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>> </property>
>>>
>>
>>
>>
>> --
>>
>>
>> Thanks
>> Devaraj K
>>
>
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
# yarn application -list
13/11/27 09:13:02 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/11/27 09:13:03 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
Total Applications:2
Application-Id Application-Name
User Queue State
Final-State Tracking-URL
application_1385514714001_0002 select count(1) from
test_carl_pageview_p(Stage-1) hive
default ACCEPTED
UNDEFINED N/A
application_1385514714001_0001 select count(1) from
test_carl_pageview_p(Stage-1) hive
default RUNNING
UNDEFINED CHBM222:59768
13/11/27 09:13:03 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
here is my capacity-scheduler.xml content
<configuration>
<property>
<name>yarn.scheduler.capacity.maximum-applications</name>
<value>10000</value>
<description>
Maximum number of applications that can be pending and running.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.3</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator</value>
<description>
The ResourceCalculator implementation to be used to compare
Resources in the scheduler.
The default i.e. DefaultResourceCalculator only uses Memory while
DominantResourceCalculator uses dominant-resource to compare
multi-dimensional resources such as Memory, CPU etc.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>default</value>
<description>
The queues at the this level (root is the root queue).
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.capacity</name>
<value>100</value>
<description>Default queue target capacity.</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
<value>0.3</value>
<description>
Default queue user limit a percentage from 0.0 to 1.0.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.minimum-user-limit-percent</name>
<value>25</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
<value>100</value>
<description>
The maximum capacity of the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.state</name>
<value>RUNNING</value>
<description>
The state of the default queue. State can be one of RUNNING or
STOPPED.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
<value>*</value>
<description>
The ACL of who can submit jobs to the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
<value>*</value>
<description>
The ACL of who can administer jobs on the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.node-locality-delay</name>
<value>-1</value>
<description>
Number of missed scheduling opportunities after which the
CapacityScheduler
attempts to schedule rack-local containers.
Typically this should be set to number of racks in the cluster, this
feature is disabled by default, set to -1.
</description>
</property>
</configuration>
On Wed, Nov 27, 2013 at 8:33 AM, ch huang <ju...@gmail.com> wrote:
> my config is
>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
>
>
> On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <de...@apache.org> wrote:
>
>> Could you check the below configuration in capacity-scheduler.xml,
>> whether is it causing to run only one AM.
>>
>> <property>
>> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
>> <value>0.1</value>
>> <description>
>> Maximum percent of resources in the cluster which can be used to
>> run
>> application masters i.e. controls number of concurrent running
>> applications.
>> </description>
>> </property>
>>
>>
>> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
>>
>>> hi,maillist:
>>> i set the following option in yarn-site.xml ,let yarn
>>> framework to use capacity scheduler,but i submit three job,only one job in
>>> running status,other two stay in accepted status,why ,the default queue
>>> only 50% capacity used,i do not know why?
>>>
>>> <property>
>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>
>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>> </property>
>>>
>>
>>
>>
>> --
>>
>>
>> Thanks
>> Devaraj K
>>
>
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
# yarn application -list
13/11/27 09:13:02 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/11/27 09:13:03 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
Total Applications:2
Application-Id Application-Name
User Queue State
Final-State Tracking-URL
application_1385514714001_0002 select count(1) from
test_carl_pageview_p(Stage-1) hive
default ACCEPTED
UNDEFINED N/A
application_1385514714001_0001 select count(1) from
test_carl_pageview_p(Stage-1) hive
default RUNNING
UNDEFINED CHBM222:59768
13/11/27 09:13:03 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
here is my capacity-scheduler.xml content
<configuration>
<property>
<name>yarn.scheduler.capacity.maximum-applications</name>
<value>10000</value>
<description>
Maximum number of applications that can be pending and running.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.3</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator</value>
<description>
The ResourceCalculator implementation to be used to compare
Resources in the scheduler.
The default i.e. DefaultResourceCalculator only uses Memory while
DominantResourceCalculator uses dominant-resource to compare
multi-dimensional resources such as Memory, CPU etc.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>default</value>
<description>
The queues at the this level (root is the root queue).
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.capacity</name>
<value>100</value>
<description>Default queue target capacity.</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
<value>0.3</value>
<description>
Default queue user limit a percentage from 0.0 to 1.0.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.minimum-user-limit-percent</name>
<value>25</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
<value>100</value>
<description>
The maximum capacity of the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.state</name>
<value>RUNNING</value>
<description>
The state of the default queue. State can be one of RUNNING or
STOPPED.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
<value>*</value>
<description>
The ACL of who can submit jobs to the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
<value>*</value>
<description>
The ACL of who can administer jobs on the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.node-locality-delay</name>
<value>-1</value>
<description>
Number of missed scheduling opportunities after which the
CapacityScheduler
attempts to schedule rack-local containers.
Typically this should be set to number of racks in the cluster, this
feature is disabled by default, set to -1.
</description>
</property>
</configuration>
On Wed, Nov 27, 2013 at 8:33 AM, ch huang <ju...@gmail.com> wrote:
> my config is
>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
>
>
> On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <de...@apache.org> wrote:
>
>> Could you check the below configuration in capacity-scheduler.xml,
>> whether is it causing to run only one AM.
>>
>> <property>
>> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
>> <value>0.1</value>
>> <description>
>> Maximum percent of resources in the cluster which can be used to
>> run
>> application masters i.e. controls number of concurrent running
>> applications.
>> </description>
>> </property>
>>
>>
>> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
>>
>>> hi,maillist:
>>> i set the following option in yarn-site.xml ,let yarn
>>> framework to use capacity scheduler,but i submit three job,only one job in
>>> running status,other two stay in accepted status,why ,the default queue
>>> only 50% capacity used,i do not know why?
>>>
>>> <property>
>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>
>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>> </property>
>>>
>>
>>
>>
>> --
>>
>>
>> Thanks
>> Devaraj K
>>
>
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
my config is
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.1</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <de...@apache.org> wrote:
> Could you check the below configuration in capacity-scheduler.xml,
> whether is it causing to run only one AM.
>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
>
>
> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i set the following option in yarn-site.xml ,let yarn
>> framework to use capacity scheduler,but i submit three job,only one job in
>> running status,other two stay in accepted status,why ,the default queue
>> only 50% capacity used,i do not know why?
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>> </property>
>>
>
>
>
> --
>
>
> Thanks
> Devaraj K
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
my config is
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.1</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <de...@apache.org> wrote:
> Could you check the below configuration in capacity-scheduler.xml,
> whether is it causing to run only one AM.
>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
>
>
> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i set the following option in yarn-site.xml ,let yarn
>> framework to use capacity scheduler,but i submit three job,only one job in
>> running status,other two stay in accepted status,why ,the default queue
>> only 50% capacity used,i do not know why?
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>> </property>
>>
>
>
>
> --
>
>
> Thanks
> Devaraj K
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
my config is
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.1</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <de...@apache.org> wrote:
> Could you check the below configuration in capacity-scheduler.xml,
> whether is it causing to run only one AM.
>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
>
>
> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i set the following option in yarn-site.xml ,let yarn
>> framework to use capacity scheduler,but i submit three job,only one job in
>> running status,other two stay in accepted status,why ,the default queue
>> only 50% capacity used,i do not know why?
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>> </property>
>>
>
>
>
> --
>
>
> Thanks
> Devaraj K
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
my config is
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.1</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <de...@apache.org> wrote:
> Could you check the below configuration in capacity-scheduler.xml,
> whether is it causing to run only one AM.
>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
>
>
> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i set the following option in yarn-site.xml ,let yarn
>> framework to use capacity scheduler,but i submit three job,only one job in
>> running status,other two stay in accepted status,why ,the default queue
>> only 50% capacity used,i do not know why?
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>> </property>
>>
>
>
>
> --
>
>
> Thanks
> Devaraj K
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
yes yarn.scheduler.capacity.maximum-am-resource-percent option is a limit,
i increase the value from 0.1 to 0.8 but ,it can still allow 2 job run
parallel,if i laugh
third job ,it still need wait,why? the whole cluster only can run 2 job
parallel?
On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <de...@apache.org> wrote:
> Could you check the below configuration in capacity-scheduler.xml,
> whether is it causing to run only one AM.
>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
>
>
> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i set the following option in yarn-site.xml ,let yarn
>> framework to use capacity scheduler,but i submit three job,only one job in
>> running status,other two stay in accepted status,why ,the default queue
>> only 50% capacity used,i do not know why?
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>> </property>
>>
>
>
>
> --
>
>
> Thanks
> Devaraj K
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
yes yarn.scheduler.capacity.maximum-am-resource-percent option is a limit,
i increase the value from 0.1 to 0.8 but ,it can still allow 2 job run
parallel,if i laugh
third job ,it still need wait,why? the whole cluster only can run 2 job
parallel?
On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <de...@apache.org> wrote:
> Could you check the below configuration in capacity-scheduler.xml,
> whether is it causing to run only one AM.
>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
>
>
> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i set the following option in yarn-site.xml ,let yarn
>> framework to use capacity scheduler,but i submit three job,only one job in
>> running status,other two stay in accepted status,why ,the default queue
>> only 50% capacity used,i do not know why?
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>> </property>
>>
>
>
>
> --
>
>
> Thanks
> Devaraj K
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
yes yarn.scheduler.capacity.maximum-am-resource-percent option is a limit,
i increase the value from 0.1 to 0.8 but ,it can still allow 2 job run
parallel,if i laugh
third job ,it still need wait,why? the whole cluster only can run 2 job
parallel?
On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <de...@apache.org> wrote:
> Could you check the below configuration in capacity-scheduler.xml,
> whether is it causing to run only one AM.
>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
>
>
> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i set the following option in yarn-site.xml ,let yarn
>> framework to use capacity scheduler,but i submit three job,only one job in
>> running status,other two stay in accepted status,why ,the default queue
>> only 50% capacity used,i do not know why?
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>> </property>
>>
>
>
>
> --
>
>
> Thanks
> Devaraj K
>
Re: default capacity scheduler only one job in running status
Posted by Devaraj K <de...@apache.org>.
Could you check the below configuration in capacity-scheduler.xml, whether
is it causing to run only one AM.
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.1</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i set the following option in yarn-site.xml ,let yarn
> framework to use capacity scheduler,but i submit three job,only one job in
> running status,other two stay in accepted status,why ,the default queue
> only 50% capacity used,i do not know why?
>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
> </property>
>
--
Thanks
Devaraj K
Re: default capacity scheduler only one job in running status
Posted by Jian He <jh...@hortonworks.com>.
yarn.scheduler.capacity.maximum-am-resource-percent limits the max number
of concurrently running AMs.
By that I mean the max memory allowed for allocating AM (defined by this
property) divided by per AM memory usage , equals to the max number of
concurrently running AMs. You may also need to check the per AM memory
configurations.
Other than that, you can also check : the queue capacity and max-capacity,
user-limit-factor. Also note how many users are using this queue.
Thanks,
Jian
On Tue, Nov 26, 2013 at 6:02 PM, ch huang <ju...@gmail.com> wrote:
> and i did some test,the following is result, i also notice that the user
> only use 11G memory,due to the resource-calculator is based on memory ,i
> guess if i let allocate more
> memory ,it can allow more job run parallel, the Total memory list in my
> http://RM_IP:8088 <http://rm_ip:8088/>
> is 24G ,i wander why the user can not take all 24G memory for running it's
> jobs? any options limit?
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.8
> yarn.scheduler.capacity.root.default.user-limit-factor 0.3
> 2 job running (small memory)
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.8
> yarn.scheduler.capacity.root.default.user-limit-factor 0.2
> 1 job running(small memory)
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.9
> yarn.scheduler.capacity.root.default.user-limit-factor 0.4
> 3 job running (small memory require job)
> 1 job running (large memory require job),3 job blocked
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.9
> yarn.scheduler.capacity.root.default.user-limit-factor 0.2
> 2 job runnning
>
>
> On Tue, Nov 26, 2013 at 6:58 PM, Olivier Renault <orenault@hortonworks.com
> > wrote:
>
>> At the queue level, you've define a certain amount of ressources. For
>> argument sake, let's say that your queue is allowed to consume 50% of your
>> cluster and max 100%. As a single user, you won't be able to consume more
>> than 50%. If you've got two different user within the queue, they would be
>> able to use 100% of teh overall cluster. You can define how much a user is
>> entitle to take of the overall Q by playing with yarn.scheduler.
>> capacity.root.production.user-limit-factor.
>>
>> If with job1 userA has reached the max he is entitled, he will need to
>> wait for some slots to become free before job2 start.
>>
>> Olivier
>>
>>
>> On 26 November 2013 10:46, ch huang <ju...@gmail.com> wrote:
>>
>>> so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
>>> modified the option value to 5,all job will be running parallel,right?
>>>
>>>
>>> On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault <
>>> orenault@hortonworks.com> wrote:
>>>
>>>> If you're running all the job from the same user, by default, you
>>>> can't take more than the value of the queue. It can be modified by setting
>>>> the following in capacity-scheduler.xml
>>>>
>>>>
>>>> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>>>>
>>>> <value>1</value>
>>>>
>>>> Olivier
>>>>
>>>>
>>>> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>>>>
>>>>> hi,maillist:
>>>>> i set the following option in yarn-site.xml ,let yarn
>>>>> framework to use capacity scheduler,but i submit three job,only one job in
>>>>> running status,other two stay in accepted status,why ,the default queue
>>>>> only 50% capacity used,i do not know why?
>>>>>
>>>>> <property>
>>>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>>>
>>>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>>>> </property>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>> Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match
>> <http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by Jian He <jh...@hortonworks.com>.
yarn.scheduler.capacity.maximum-am-resource-percent limits the max number
of concurrently running AMs.
By that I mean the max memory allowed for allocating AM (defined by this
property) divided by per AM memory usage , equals to the max number of
concurrently running AMs. You may also need to check the per AM memory
configurations.
Other than that, you can also check : the queue capacity and max-capacity,
user-limit-factor. Also note how many users are using this queue.
Thanks,
Jian
On Tue, Nov 26, 2013 at 6:02 PM, ch huang <ju...@gmail.com> wrote:
> and i did some test,the following is result, i also notice that the user
> only use 11G memory,due to the resource-calculator is based on memory ,i
> guess if i let allocate more
> memory ,it can allow more job run parallel, the Total memory list in my
> http://RM_IP:8088 <http://rm_ip:8088/>
> is 24G ,i wander why the user can not take all 24G memory for running it's
> jobs? any options limit?
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.8
> yarn.scheduler.capacity.root.default.user-limit-factor 0.3
> 2 job running (small memory)
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.8
> yarn.scheduler.capacity.root.default.user-limit-factor 0.2
> 1 job running(small memory)
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.9
> yarn.scheduler.capacity.root.default.user-limit-factor 0.4
> 3 job running (small memory require job)
> 1 job running (large memory require job),3 job blocked
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.9
> yarn.scheduler.capacity.root.default.user-limit-factor 0.2
> 2 job runnning
>
>
> On Tue, Nov 26, 2013 at 6:58 PM, Olivier Renault <orenault@hortonworks.com
> > wrote:
>
>> At the queue level, you've define a certain amount of ressources. For
>> argument sake, let's say that your queue is allowed to consume 50% of your
>> cluster and max 100%. As a single user, you won't be able to consume more
>> than 50%. If you've got two different user within the queue, they would be
>> able to use 100% of teh overall cluster. You can define how much a user is
>> entitle to take of the overall Q by playing with yarn.scheduler.
>> capacity.root.production.user-limit-factor.
>>
>> If with job1 userA has reached the max he is entitled, he will need to
>> wait for some slots to become free before job2 start.
>>
>> Olivier
>>
>>
>> On 26 November 2013 10:46, ch huang <ju...@gmail.com> wrote:
>>
>>> so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
>>> modified the option value to 5,all job will be running parallel,right?
>>>
>>>
>>> On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault <
>>> orenault@hortonworks.com> wrote:
>>>
>>>> If you're running all the job from the same user, by default, you
>>>> can't take more than the value of the queue. It can be modified by setting
>>>> the following in capacity-scheduler.xml
>>>>
>>>>
>>>> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>>>>
>>>> <value>1</value>
>>>>
>>>> Olivier
>>>>
>>>>
>>>> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>>>>
>>>>> hi,maillist:
>>>>> i set the following option in yarn-site.xml ,let yarn
>>>>> framework to use capacity scheduler,but i submit three job,only one job in
>>>>> running status,other two stay in accepted status,why ,the default queue
>>>>> only 50% capacity used,i do not know why?
>>>>>
>>>>> <property>
>>>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>>>
>>>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>>>> </property>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>> Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match
>> <http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by Jian He <jh...@hortonworks.com>.
yarn.scheduler.capacity.maximum-am-resource-percent limits the max number
of concurrently running AMs.
By that I mean the max memory allowed for allocating AM (defined by this
property) divided by per AM memory usage , equals to the max number of
concurrently running AMs. You may also need to check the per AM memory
configurations.
Other than that, you can also check : the queue capacity and max-capacity,
user-limit-factor. Also note how many users are using this queue.
Thanks,
Jian
On Tue, Nov 26, 2013 at 6:02 PM, ch huang <ju...@gmail.com> wrote:
> and i did some test,the following is result, i also notice that the user
> only use 11G memory,due to the resource-calculator is based on memory ,i
> guess if i let allocate more
> memory ,it can allow more job run parallel, the Total memory list in my
> http://RM_IP:8088 <http://rm_ip:8088/>
> is 24G ,i wander why the user can not take all 24G memory for running it's
> jobs? any options limit?
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.8
> yarn.scheduler.capacity.root.default.user-limit-factor 0.3
> 2 job running (small memory)
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.8
> yarn.scheduler.capacity.root.default.user-limit-factor 0.2
> 1 job running(small memory)
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.9
> yarn.scheduler.capacity.root.default.user-limit-factor 0.4
> 3 job running (small memory require job)
> 1 job running (large memory require job),3 job blocked
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.9
> yarn.scheduler.capacity.root.default.user-limit-factor 0.2
> 2 job runnning
>
>
> On Tue, Nov 26, 2013 at 6:58 PM, Olivier Renault <orenault@hortonworks.com
> > wrote:
>
>> At the queue level, you've define a certain amount of ressources. For
>> argument sake, let's say that your queue is allowed to consume 50% of your
>> cluster and max 100%. As a single user, you won't be able to consume more
>> than 50%. If you've got two different user within the queue, they would be
>> able to use 100% of teh overall cluster. You can define how much a user is
>> entitle to take of the overall Q by playing with yarn.scheduler.
>> capacity.root.production.user-limit-factor.
>>
>> If with job1 userA has reached the max he is entitled, he will need to
>> wait for some slots to become free before job2 start.
>>
>> Olivier
>>
>>
>> On 26 November 2013 10:46, ch huang <ju...@gmail.com> wrote:
>>
>>> so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
>>> modified the option value to 5,all job will be running parallel,right?
>>>
>>>
>>> On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault <
>>> orenault@hortonworks.com> wrote:
>>>
>>>> If you're running all the job from the same user, by default, you
>>>> can't take more than the value of the queue. It can be modified by setting
>>>> the following in capacity-scheduler.xml
>>>>
>>>>
>>>> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>>>>
>>>> <value>1</value>
>>>>
>>>> Olivier
>>>>
>>>>
>>>> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>>>>
>>>>> hi,maillist:
>>>>> i set the following option in yarn-site.xml ,let yarn
>>>>> framework to use capacity scheduler,but i submit three job,only one job in
>>>>> running status,other two stay in accepted status,why ,the default queue
>>>>> only 50% capacity used,i do not know why?
>>>>>
>>>>> <property>
>>>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>>>
>>>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>>>> </property>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>> Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match
>> <http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by Jian He <jh...@hortonworks.com>.
yarn.scheduler.capacity.maximum-am-resource-percent limits the max number
of concurrently running AMs.
By that I mean the max memory allowed for allocating AM (defined by this
property) divided by per AM memory usage , equals to the max number of
concurrently running AMs. You may also need to check the per AM memory
configurations.
Other than that, you can also check : the queue capacity and max-capacity,
user-limit-factor. Also note how many users are using this queue.
Thanks,
Jian
On Tue, Nov 26, 2013 at 6:02 PM, ch huang <ju...@gmail.com> wrote:
> and i did some test,the following is result, i also notice that the user
> only use 11G memory,due to the resource-calculator is based on memory ,i
> guess if i let allocate more
> memory ,it can allow more job run parallel, the Total memory list in my
> http://RM_IP:8088 <http://rm_ip:8088/>
> is 24G ,i wander why the user can not take all 24G memory for running it's
> jobs? any options limit?
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.8
> yarn.scheduler.capacity.root.default.user-limit-factor 0.3
> 2 job running (small memory)
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.8
> yarn.scheduler.capacity.root.default.user-limit-factor 0.2
> 1 job running(small memory)
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.9
> yarn.scheduler.capacity.root.default.user-limit-factor 0.4
> 3 job running (small memory require job)
> 1 job running (large memory require job),3 job blocked
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent 0.9
> yarn.scheduler.capacity.root.default.user-limit-factor 0.2
> 2 job runnning
>
>
> On Tue, Nov 26, 2013 at 6:58 PM, Olivier Renault <orenault@hortonworks.com
> > wrote:
>
>> At the queue level, you've define a certain amount of ressources. For
>> argument sake, let's say that your queue is allowed to consume 50% of your
>> cluster and max 100%. As a single user, you won't be able to consume more
>> than 50%. If you've got two different user within the queue, they would be
>> able to use 100% of teh overall cluster. You can define how much a user is
>> entitle to take of the overall Q by playing with yarn.scheduler.
>> capacity.root.production.user-limit-factor.
>>
>> If with job1 userA has reached the max he is entitled, he will need to
>> wait for some slots to become free before job2 start.
>>
>> Olivier
>>
>>
>> On 26 November 2013 10:46, ch huang <ju...@gmail.com> wrote:
>>
>>> so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
>>> modified the option value to 5,all job will be running parallel,right?
>>>
>>>
>>> On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault <
>>> orenault@hortonworks.com> wrote:
>>>
>>>> If you're running all the job from the same user, by default, you
>>>> can't take more than the value of the queue. It can be modified by setting
>>>> the following in capacity-scheduler.xml
>>>>
>>>>
>>>> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>>>>
>>>> <value>1</value>
>>>>
>>>> Olivier
>>>>
>>>>
>>>> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>>>>
>>>>> hi,maillist:
>>>>> i set the following option in yarn-site.xml ,let yarn
>>>>> framework to use capacity scheduler,but i submit three job,only one job in
>>>>> running status,other two stay in accepted status,why ,the default queue
>>>>> only 50% capacity used,i do not know why?
>>>>>
>>>>> <property>
>>>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>>>
>>>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>>>> </property>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>> Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match
>> <http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
and i did some test,the following is result, i also notice that the user
only use 11G memory,due to the resource-calculator is based on memory ,i
guess if i let allocate more
memory ,it can allow more job run parallel, the Total memory list in my
http://RM_IP:8088 <http://rm_ip:8088/>
is 24G ,i wander why the user can not take all 24G memory for running it's
jobs? any options limit?
yarn.scheduler.capacity.maximum-am-resource-percent 0.8
yarn.scheduler.capacity.root.default.user-limit-factor 0.3
2 job running (small memory)
yarn.scheduler.capacity.maximum-am-resource-percent 0.8
yarn.scheduler.capacity.root.default.user-limit-factor 0.2
1 job running(small memory)
yarn.scheduler.capacity.maximum-am-resource-percent 0.9
yarn.scheduler.capacity.root.default.user-limit-factor 0.4
3 job running (small memory require job)
1 job running (large memory require job),3 job blocked
yarn.scheduler.capacity.maximum-am-resource-percent 0.9
yarn.scheduler.capacity.root.default.user-limit-factor 0.2
2 job runnning
On Tue, Nov 26, 2013 at 6:58 PM, Olivier Renault
<or...@hortonworks.com>wrote:
> At the queue level, you've define a certain amount of ressources. For
> argument sake, let's say that your queue is allowed to consume 50% of your
> cluster and max 100%. As a single user, you won't be able to consume more
> than 50%. If you've got two different user within the queue, they would be
> able to use 100% of teh overall cluster. You can define how much a user is
> entitle to take of the overall Q by playing with yarn.scheduler.
> capacity.root.production.user-limit-factor.
>
> If with job1 userA has reached the max he is entitled, he will need to
> wait for some slots to become free before job2 start.
>
> Olivier
>
>
> On 26 November 2013 10:46, ch huang <ju...@gmail.com> wrote:
>
>> so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
>> modified the option value to 5,all job will be running parallel,right?
>>
>>
>> On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault <
>> orenault@hortonworks.com> wrote:
>>
>>> If you're running all the job from the same user, by default, you
>>> can't take more than the value of the queue. It can be modified by setting
>>> the following in capacity-scheduler.xml
>>>
>>> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>>>
>>> <value>1</value>
>>>
>>> Olivier
>>>
>>>
>>> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>>>
>>>> hi,maillist:
>>>> i set the following option in yarn-site.xml ,let yarn
>>>> framework to use capacity scheduler,but i submit three job,only one job in
>>>> running status,other two stay in accepted status,why ,the default queue
>>>> only 50% capacity used,i do not know why?
>>>>
>>>> <property>
>>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>>
>>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>>> </property>
>>>>
>>>
>>>
>>>
>>>
>>>
> Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match
> <http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
and i did some test,the following is result, i also notice that the user
only use 11G memory,due to the resource-calculator is based on memory ,i
guess if i let allocate more
memory ,it can allow more job run parallel, the Total memory list in my
http://RM_IP:8088 <http://rm_ip:8088/>
is 24G ,i wander why the user can not take all 24G memory for running it's
jobs? any options limit?
yarn.scheduler.capacity.maximum-am-resource-percent 0.8
yarn.scheduler.capacity.root.default.user-limit-factor 0.3
2 job running (small memory)
yarn.scheduler.capacity.maximum-am-resource-percent 0.8
yarn.scheduler.capacity.root.default.user-limit-factor 0.2
1 job running(small memory)
yarn.scheduler.capacity.maximum-am-resource-percent 0.9
yarn.scheduler.capacity.root.default.user-limit-factor 0.4
3 job running (small memory require job)
1 job running (large memory require job),3 job blocked
yarn.scheduler.capacity.maximum-am-resource-percent 0.9
yarn.scheduler.capacity.root.default.user-limit-factor 0.2
2 job runnning
On Tue, Nov 26, 2013 at 6:58 PM, Olivier Renault
<or...@hortonworks.com>wrote:
> At the queue level, you've define a certain amount of ressources. For
> argument sake, let's say that your queue is allowed to consume 50% of your
> cluster and max 100%. As a single user, you won't be able to consume more
> than 50%. If you've got two different user within the queue, they would be
> able to use 100% of teh overall cluster. You can define how much a user is
> entitle to take of the overall Q by playing with yarn.scheduler.
> capacity.root.production.user-limit-factor.
>
> If with job1 userA has reached the max he is entitled, he will need to
> wait for some slots to become free before job2 start.
>
> Olivier
>
>
> On 26 November 2013 10:46, ch huang <ju...@gmail.com> wrote:
>
>> so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
>> modified the option value to 5,all job will be running parallel,right?
>>
>>
>> On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault <
>> orenault@hortonworks.com> wrote:
>>
>>> If you're running all the job from the same user, by default, you
>>> can't take more than the value of the queue. It can be modified by setting
>>> the following in capacity-scheduler.xml
>>>
>>> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>>>
>>> <value>1</value>
>>>
>>> Olivier
>>>
>>>
>>> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>>>
>>>> hi,maillist:
>>>> i set the following option in yarn-site.xml ,let yarn
>>>> framework to use capacity scheduler,but i submit three job,only one job in
>>>> running status,other two stay in accepted status,why ,the default queue
>>>> only 50% capacity used,i do not know why?
>>>>
>>>> <property>
>>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>>
>>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>>> </property>
>>>>
>>>
>>>
>>>
>>>
>>>
> Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match
> <http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
and i did some test,the following is result, i also notice that the user
only use 11G memory,due to the resource-calculator is based on memory ,i
guess if i let allocate more
memory ,it can allow more job run parallel, the Total memory list in my
http://RM_IP:8088 <http://rm_ip:8088/>
is 24G ,i wander why the user can not take all 24G memory for running it's
jobs? any options limit?
yarn.scheduler.capacity.maximum-am-resource-percent 0.8
yarn.scheduler.capacity.root.default.user-limit-factor 0.3
2 job running (small memory)
yarn.scheduler.capacity.maximum-am-resource-percent 0.8
yarn.scheduler.capacity.root.default.user-limit-factor 0.2
1 job running(small memory)
yarn.scheduler.capacity.maximum-am-resource-percent 0.9
yarn.scheduler.capacity.root.default.user-limit-factor 0.4
3 job running (small memory require job)
1 job running (large memory require job),3 job blocked
yarn.scheduler.capacity.maximum-am-resource-percent 0.9
yarn.scheduler.capacity.root.default.user-limit-factor 0.2
2 job runnning
On Tue, Nov 26, 2013 at 6:58 PM, Olivier Renault
<or...@hortonworks.com>wrote:
> At the queue level, you've define a certain amount of ressources. For
> argument sake, let's say that your queue is allowed to consume 50% of your
> cluster and max 100%. As a single user, you won't be able to consume more
> than 50%. If you've got two different user within the queue, they would be
> able to use 100% of teh overall cluster. You can define how much a user is
> entitle to take of the overall Q by playing with yarn.scheduler.
> capacity.root.production.user-limit-factor.
>
> If with job1 userA has reached the max he is entitled, he will need to
> wait for some slots to become free before job2 start.
>
> Olivier
>
>
> On 26 November 2013 10:46, ch huang <ju...@gmail.com> wrote:
>
>> so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
>> modified the option value to 5,all job will be running parallel,right?
>>
>>
>> On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault <
>> orenault@hortonworks.com> wrote:
>>
>>> If you're running all the job from the same user, by default, you
>>> can't take more than the value of the queue. It can be modified by setting
>>> the following in capacity-scheduler.xml
>>>
>>> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>>>
>>> <value>1</value>
>>>
>>> Olivier
>>>
>>>
>>> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>>>
>>>> hi,maillist:
>>>> i set the following option in yarn-site.xml ,let yarn
>>>> framework to use capacity scheduler,but i submit three job,only one job in
>>>> running status,other two stay in accepted status,why ,the default queue
>>>> only 50% capacity used,i do not know why?
>>>>
>>>> <property>
>>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>>
>>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>>> </property>
>>>>
>>>
>>>
>>>
>>>
>>>
> Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match
> <http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
and i did some test,the following is result, i also notice that the user
only use 11G memory,due to the resource-calculator is based on memory ,i
guess if i let allocate more
memory ,it can allow more job run parallel, the Total memory list in my
http://RM_IP:8088 <http://rm_ip:8088/>
is 24G ,i wander why the user can not take all 24G memory for running it's
jobs? any options limit?
yarn.scheduler.capacity.maximum-am-resource-percent 0.8
yarn.scheduler.capacity.root.default.user-limit-factor 0.3
2 job running (small memory)
yarn.scheduler.capacity.maximum-am-resource-percent 0.8
yarn.scheduler.capacity.root.default.user-limit-factor 0.2
1 job running(small memory)
yarn.scheduler.capacity.maximum-am-resource-percent 0.9
yarn.scheduler.capacity.root.default.user-limit-factor 0.4
3 job running (small memory require job)
1 job running (large memory require job),3 job blocked
yarn.scheduler.capacity.maximum-am-resource-percent 0.9
yarn.scheduler.capacity.root.default.user-limit-factor 0.2
2 job runnning
On Tue, Nov 26, 2013 at 6:58 PM, Olivier Renault
<or...@hortonworks.com>wrote:
> At the queue level, you've define a certain amount of ressources. For
> argument sake, let's say that your queue is allowed to consume 50% of your
> cluster and max 100%. As a single user, you won't be able to consume more
> than 50%. If you've got two different user within the queue, they would be
> able to use 100% of teh overall cluster. You can define how much a user is
> entitle to take of the overall Q by playing with yarn.scheduler.
> capacity.root.production.user-limit-factor.
>
> If with job1 userA has reached the max he is entitled, he will need to
> wait for some slots to become free before job2 start.
>
> Olivier
>
>
> On 26 November 2013 10:46, ch huang <ju...@gmail.com> wrote:
>
>> so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
>> modified the option value to 5,all job will be running parallel,right?
>>
>>
>> On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault <
>> orenault@hortonworks.com> wrote:
>>
>>> If you're running all the job from the same user, by default, you
>>> can't take more than the value of the queue. It can be modified by setting
>>> the following in capacity-scheduler.xml
>>>
>>> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>>>
>>> <value>1</value>
>>>
>>> Olivier
>>>
>>>
>>> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>>>
>>>> hi,maillist:
>>>> i set the following option in yarn-site.xml ,let yarn
>>>> framework to use capacity scheduler,but i submit three job,only one job in
>>>> running status,other two stay in accepted status,why ,the default queue
>>>> only 50% capacity used,i do not know why?
>>>>
>>>> <property>
>>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>>
>>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>>> </property>
>>>>
>>>
>>>
>>>
>>>
>>>
> Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match
> <http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: default capacity scheduler only one job in running status
Posted by Olivier Renault <or...@hortonworks.com>.
At the queue level, you've define a certain amount of ressources. For
argument sake, let's say that your queue is allowed to consume 50% of your
cluster and max 100%. As a single user, you won't be able to consume more
than 50%. If you've got two different user within the queue, they would be
able to use 100% of teh overall cluster. You can define how much a user is
entitle to take of the overall Q by playing with yarn.scheduler.
capacity.root.production.user-limit-factor.
If with job1 userA has reached the max he is entitled, he will need to wait
for some slots to become free before job2 start.
Olivier
On 26 November 2013 10:46, ch huang <ju...@gmail.com> wrote:
> so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
> modified the option value to 5,all job will be running parallel,right?
>
>
> On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault <orenault@hortonworks.com
> > wrote:
>
>> If you're running all the job from the same user, by default, you can't
>> take more than the value of the queue. It can be modified by setting the
>> following in capacity-scheduler.xml
>>
>> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>>
>> <value>1</value>
>>
>> Olivier
>>
>>
>> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>>
>>> hi,maillist:
>>> i set the following option in yarn-site.xml ,let yarn
>>> framework to use capacity scheduler,but i submit three job,only one job in
>>> running status,other two stay in accepted status,why ,the default queue
>>> only 50% capacity used,i do not know why?
>>>
>>> <property>
>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>
>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>> </property>
>>>
>>
>>
>>
>>
>>
Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match
<http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by Olivier Renault <or...@hortonworks.com>.
At the queue level, you've define a certain amount of ressources. For
argument sake, let's say that your queue is allowed to consume 50% of your
cluster and max 100%. As a single user, you won't be able to consume more
than 50%. If you've got two different user within the queue, they would be
able to use 100% of teh overall cluster. You can define how much a user is
entitle to take of the overall Q by playing with yarn.scheduler.
capacity.root.production.user-limit-factor.
If with job1 userA has reached the max he is entitled, he will need to wait
for some slots to become free before job2 start.
Olivier
On 26 November 2013 10:46, ch huang <ju...@gmail.com> wrote:
> so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
> modified the option value to 5,all job will be running parallel,right?
>
>
> On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault <orenault@hortonworks.com
> > wrote:
>
>> If you're running all the job from the same user, by default, you can't
>> take more than the value of the queue. It can be modified by setting the
>> following in capacity-scheduler.xml
>>
>> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>>
>> <value>1</value>
>>
>> Olivier
>>
>>
>> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>>
>>> hi,maillist:
>>> i set the following option in yarn-site.xml ,let yarn
>>> framework to use capacity scheduler,but i submit three job,only one job in
>>> running status,other two stay in accepted status,why ,the default queue
>>> only 50% capacity used,i do not know why?
>>>
>>> <property>
>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>
>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>> </property>
>>>
>>
>>
>>
>>
>>
Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match
<http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by Olivier Renault <or...@hortonworks.com>.
At the queue level, you've define a certain amount of ressources. For
argument sake, let's say that your queue is allowed to consume 50% of your
cluster and max 100%. As a single user, you won't be able to consume more
than 50%. If you've got two different user within the queue, they would be
able to use 100% of teh overall cluster. You can define how much a user is
entitle to take of the overall Q by playing with yarn.scheduler.
capacity.root.production.user-limit-factor.
If with job1 userA has reached the max he is entitled, he will need to wait
for some slots to become free before job2 start.
Olivier
On 26 November 2013 10:46, ch huang <ju...@gmail.com> wrote:
> so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
> modified the option value to 5,all job will be running parallel,right?
>
>
> On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault <orenault@hortonworks.com
> > wrote:
>
>> If you're running all the job from the same user, by default, you can't
>> take more than the value of the queue. It can be modified by setting the
>> following in capacity-scheduler.xml
>>
>> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>>
>> <value>1</value>
>>
>> Olivier
>>
>>
>> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>>
>>> hi,maillist:
>>> i set the following option in yarn-site.xml ,let yarn
>>> framework to use capacity scheduler,but i submit three job,only one job in
>>> running status,other two stay in accepted status,why ,the default queue
>>> only 50% capacity used,i do not know why?
>>>
>>> <property>
>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>
>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>> </property>
>>>
>>
>>
>>
>>
>>
Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match
<http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by Olivier Renault <or...@hortonworks.com>.
At the queue level, you've define a certain amount of ressources. For
argument sake, let's say that your queue is allowed to consume 50% of your
cluster and max 100%. As a single user, you won't be able to consume more
than 50%. If you've got two different user within the queue, they would be
able to use 100% of teh overall cluster. You can define how much a user is
entitle to take of the overall Q by playing with yarn.scheduler.
capacity.root.production.user-limit-factor.
If with job1 userA has reached the max he is entitled, he will need to wait
for some slots to become free before job2 start.
Olivier
On 26 November 2013 10:46, ch huang <ju...@gmail.com> wrote:
> so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
> modified the option value to 5,all job will be running parallel,right?
>
>
> On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault <orenault@hortonworks.com
> > wrote:
>
>> If you're running all the job from the same user, by default, you can't
>> take more than the value of the queue. It can be modified by setting the
>> following in capacity-scheduler.xml
>>
>> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>>
>> <value>1</value>
>>
>> Olivier
>>
>>
>> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>>
>>> hi,maillist:
>>> i set the following option in yarn-site.xml ,let yarn
>>> framework to use capacity scheduler,but i submit three job,only one job in
>>> running status,other two stay in accepted status,why ,the default queue
>>> only 50% capacity used,i do not know why?
>>>
>>> <property>
>>> <name>yarn.resourcemanager.scheduler.class</name>
>>>
>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>> </property>
>>>
>>
>>
>>
>>
>>
Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match
<http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
modified the option value to 5,all job will be running parallel,right?
On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault
<or...@hortonworks.com>wrote:
> If you're running all the job from the same user, by default, you can't
> take more than the value of the queue. It can be modified by setting the
> following in capacity-scheduler.xml
>
> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>
> <value>1</value>
>
> Olivier
>
>
> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i set the following option in yarn-site.xml ,let yarn
>> framework to use capacity scheduler,but i submit three job,only one job in
>> running status,other two stay in accepted status,why ,the default queue
>> only 50% capacity used,i do not know why?
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>> </property>
>>
>
>
>
> --
> * Olivier Renault * Solution Engineer
> ------------------------------
>
> Phone: +44 7500 933 036
> Email: orenault@hortonworks.com
> Website: http://www.hortonworks.com/
>
> * Follow Us: *
> <http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> [image: photo]
>
> Latest From Our Blog: Fight Fraud with Big Data Analytics <http://hortonworks.com/blog/fight-fraud-with-big-data-analytics/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
modified the option value to 5,all job will be running parallel,right?
On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault
<or...@hortonworks.com>wrote:
> If you're running all the job from the same user, by default, you can't
> take more than the value of the queue. It can be modified by setting the
> following in capacity-scheduler.xml
>
> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>
> <value>1</value>
>
> Olivier
>
>
> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i set the following option in yarn-site.xml ,let yarn
>> framework to use capacity scheduler,but i submit three job,only one job in
>> running status,other two stay in accepted status,why ,the default queue
>> only 50% capacity used,i do not know why?
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>> </property>
>>
>
>
>
> --
> * Olivier Renault * Solution Engineer
> ------------------------------
>
> Phone: +44 7500 933 036
> Email: orenault@hortonworks.com
> Website: http://www.hortonworks.com/
>
> * Follow Us: *
> <http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> [image: photo]
>
> Latest From Our Blog: Fight Fraud with Big Data Analytics <http://hortonworks.com/blog/fight-fraud-with-big-data-analytics/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
modified the option value to 5,all job will be running parallel,right?
On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault
<or...@hortonworks.com>wrote:
> If you're running all the job from the same user, by default, you can't
> take more than the value of the queue. It can be modified by setting the
> following in capacity-scheduler.xml
>
> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>
> <value>1</value>
>
> Olivier
>
>
> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i set the following option in yarn-site.xml ,let yarn
>> framework to use capacity scheduler,but i submit three job,only one job in
>> running status,other two stay in accepted status,why ,the default queue
>> only 50% capacity used,i do not know why?
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>> </property>
>>
>
>
>
> --
> * Olivier Renault * Solution Engineer
> ------------------------------
>
> Phone: +44 7500 933 036
> Email: orenault@hortonworks.com
> Website: http://www.hortonworks.com/
>
> * Follow Us: *
> <http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> [image: photo]
>
> Latest From Our Blog: Fight Fraud with Big Data Analytics <http://hortonworks.com/blog/fight-fraud-with-big-data-analytics/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by ch huang <ju...@gmail.com>.
so ,by default ,user A submitted 5 jobs ,only 1 job is running ,if i
modified the option value to 5,all job will be running parallel,right?
On Tue, Nov 26, 2013 at 6:29 PM, Olivier Renault
<or...@hortonworks.com>wrote:
> If you're running all the job from the same user, by default, you can't
> take more than the value of the queue. It can be modified by setting the
> following in capacity-scheduler.xml
>
> <name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
>
> <value>1</value>
>
> Olivier
>
>
> On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i set the following option in yarn-site.xml ,let yarn
>> framework to use capacity scheduler,but i submit three job,only one job in
>> running status,other two stay in accepted status,why ,the default queue
>> only 50% capacity used,i do not know why?
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>> </property>
>>
>
>
>
> --
> * Olivier Renault * Solution Engineer
> ------------------------------
>
> Phone: +44 7500 933 036
> Email: orenault@hortonworks.com
> Website: http://www.hortonworks.com/
>
> * Follow Us: *
> <http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> [image: photo]
>
> Latest From Our Blog: Fight Fraud with Big Data Analytics <http://hortonworks.com/blog/fight-fraud-with-big-data-analytics/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by Olivier Renault <or...@hortonworks.com>.
If you're running all the job from the same user, by default, you can't
take more than the value of the queue. It can be modified by setting the
following in capacity-scheduler.xml
<name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
<value>1</value>
Olivier
On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i set the following option in yarn-site.xml ,let yarn
> framework to use capacity scheduler,but i submit three job,only one job in
> running status,other two stay in accepted status,why ,the default queue
> only 50% capacity used,i do not know why?
>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
> </property>
>
--
* Olivier Renault * Solution Engineer
------------------------------
Phone: +44 7500 933 036
Email: orenault@hortonworks.com
Website: http://www.hortonworks.com/
* Follow Us: *
<http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
[image: photo]
Latest From Our Blog: Fight Fraud with Big Data Analytics
<http://hortonworks.com/blog/fight-fraud-with-big-data-analytics/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by Olivier Renault <or...@hortonworks.com>.
If you're running all the job from the same user, by default, you can't
take more than the value of the queue. It can be modified by setting the
following in capacity-scheduler.xml
<name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
<value>1</value>
Olivier
On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i set the following option in yarn-site.xml ,let yarn
> framework to use capacity scheduler,but i submit three job,only one job in
> running status,other two stay in accepted status,why ,the default queue
> only 50% capacity used,i do not know why?
>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
> </property>
>
--
* Olivier Renault * Solution Engineer
------------------------------
Phone: +44 7500 933 036
Email: orenault@hortonworks.com
Website: http://www.hortonworks.com/
* Follow Us: *
<http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
[image: photo]
Latest From Our Blog: Fight Fraud with Big Data Analytics
<http://hortonworks.com/blog/fight-fraud-with-big-data-analytics/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by Olivier Renault <or...@hortonworks.com>.
If you're running all the job from the same user, by default, you can't
take more than the value of the queue. It can be modified by setting the
following in capacity-scheduler.xml
<name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
<value>1</value>
Olivier
On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i set the following option in yarn-site.xml ,let yarn
> framework to use capacity scheduler,but i submit three job,only one job in
> running status,other two stay in accepted status,why ,the default queue
> only 50% capacity used,i do not know why?
>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
> </property>
>
--
* Olivier Renault * Solution Engineer
------------------------------
Phone: +44 7500 933 036
Email: orenault@hortonworks.com
Website: http://www.hortonworks.com/
* Follow Us: *
<http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
[image: photo]
Latest From Our Blog: Fight Fraud with Big Data Analytics
<http://hortonworks.com/blog/fight-fraud-with-big-data-analytics/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by Devaraj K <de...@apache.org>.
Could you check the below configuration in capacity-scheduler.xml, whether
is it causing to run only one AM.
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.1</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i set the following option in yarn-site.xml ,let yarn
> framework to use capacity scheduler,but i submit three job,only one job in
> running status,other two stay in accepted status,why ,the default queue
> only 50% capacity used,i do not know why?
>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
> </property>
>
--
Thanks
Devaraj K
Re: default capacity scheduler only one job in running status
Posted by Devaraj K <de...@apache.org>.
Could you check the below configuration in capacity-scheduler.xml, whether
is it causing to run only one AM.
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.1</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i set the following option in yarn-site.xml ,let yarn
> framework to use capacity scheduler,but i submit three job,only one job in
> running status,other two stay in accepted status,why ,the default queue
> only 50% capacity used,i do not know why?
>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
> </property>
>
--
Thanks
Devaraj K
Re: default capacity scheduler only one job in running status
Posted by Olivier Renault <or...@hortonworks.com>.
If you're running all the job from the same user, by default, you can't
take more than the value of the queue. It can be modified by setting the
following in capacity-scheduler.xml
<name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
<value>1</value>
Olivier
On 26 November 2013 09:20, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i set the following option in yarn-site.xml ,let yarn
> framework to use capacity scheduler,but i submit three job,only one job in
> running status,other two stay in accepted status,why ,the default queue
> only 50% capacity used,i do not know why?
>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
> </property>
>
--
* Olivier Renault * Solution Engineer
------------------------------
Phone: +44 7500 933 036
Email: orenault@hortonworks.com
Website: http://www.hortonworks.com/
* Follow Us: *
<http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
[image: photo]
Latest From Our Blog: Fight Fraud with Big Data Analytics
<http://hortonworks.com/blog/fight-fraud-with-big-data-analytics/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: default capacity scheduler only one job in running status
Posted by Devaraj K <de...@apache.org>.
Could you check the below configuration in capacity-scheduler.xml, whether
is it causing to run only one AM.
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.1</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
On Tue, Nov 26, 2013 at 2:50 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i set the following option in yarn-site.xml ,let yarn
> framework to use capacity scheduler,but i submit three job,only one job in
> running status,other two stay in accepted status,why ,the default queue
> only 50% capacity used,i do not know why?
>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
> </property>
>
--
Thanks
Devaraj K