You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ivan Tretyakov <it...@griddynamics.com> on 2013/11/19 12:26:45 UTC
Limit on total jobs running using fair scheduler
Hello!
We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
We need to limit total number of jobs which can run at the same time on
cluster.
I can see maxRunningJobs options but it sets limit for pool or user.
We wouldn't like to limit each pool or user we just need to set limit on
total number of jobs running.
Is it possible to do it using fair scheduler?
Can capacity scheduler help here?
Maybe there are other options to achieve the goal?
Thanks in advance!
--
Best Regards
Ivan Tretyakov
Re: Limit on total jobs running using fair scheduler
Posted by Azuryy Yu <az...@gmail.com>.
fot MRv1, Its impossible.
On 2013-11-22 5:46 AM, "Ivan Tretyakov" <it...@griddynamics.com>
wrote:
> Thank you for your replies!
>
> We are using MR version 1 and my question is regarding this version.
>
> Omkar,
> are you talking about MR1 or MR2? I didn't find property to limit number
> of running jobs per queue for capacity scheduler using MR1. Did I wrong
> somewhere? What exactly option do you mean?
>
> Sandy,
> thanks, I got it. But unfortunately we are using MR1 for now.
>
>
> On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Unfortunately, this is not possible in the MR1 fair scheduler without
>> setting the jobs for individual pools. In MR2, fair scheduler hierarchical
>> queues will allow setting maxRunningApps at the top of the hierarchy, which
>> would have the effect you're looking for.
>>
>> -Sandy
>>
>>
>> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>>
>>> Not sure about the fair scheduler but in capacity scheduler you can
>>> achieve this by controlling number of jobs/applications per queue.
>>>
>>> Thanks,
>>> Omkar Joshi
>>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>>
>>>
>>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>>> itretyakov@griddynamics.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>>> We need to limit total number of jobs which can run at the same time on
>>>> cluster.
>>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>>> We wouldn't like to limit each pool or user we just need to set limit
>>>> on total number of jobs running.
>>>>
>>>> Is it possible to do it using fair scheduler?
>>>> Can capacity scheduler help here?
>>>> Maybe there are other options to achieve the goal?
>>>>
>>>> Thanks in advance!
>>>>
>>>> --
>>>> Best Regards
>>>> Ivan Tretyakov
>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>
Re: Limit on total jobs running using fair scheduler
Posted by Omkar Joshi <oj...@hortonworks.com>.
MRv2 :)
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Thu, Nov 21, 2013 at 1:46 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:
> Thank you for your replies!
>
> We are using MR version 1 and my question is regarding this version.
>
> Omkar,
> are you talking about MR1 or MR2? I didn't find property to limit number
> of running jobs per queue for capacity scheduler using MR1. Did I wrong
> somewhere? What exactly option do you mean?
>
> Sandy,
> thanks, I got it. But unfortunately we are using MR1 for now.
>
>
> On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Unfortunately, this is not possible in the MR1 fair scheduler without
>> setting the jobs for individual pools. In MR2, fair scheduler hierarchical
>> queues will allow setting maxRunningApps at the top of the hierarchy, which
>> would have the effect you're looking for.
>>
>> -Sandy
>>
>>
>> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>>
>>> Not sure about the fair scheduler but in capacity scheduler you can
>>> achieve this by controlling number of jobs/applications per queue.
>>>
>>> Thanks,
>>> Omkar Joshi
>>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>>
>>>
>>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>>> itretyakov@griddynamics.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>>> We need to limit total number of jobs which can run at the same time on
>>>> cluster.
>>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>>> We wouldn't like to limit each pool or user we just need to set limit
>>>> on total number of jobs running.
>>>>
>>>> Is it possible to do it using fair scheduler?
>>>> Can capacity scheduler help here?
>>>> Maybe there are other options to achieve the goal?
>>>>
>>>> Thanks in advance!
>>>>
>>>> --
>>>> Best Regards
>>>> Ivan Tretyakov
>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Limit on total jobs running using fair scheduler
Posted by Azuryy Yu <az...@gmail.com>.
fot MRv1, Its impossible.
On 2013-11-22 5:46 AM, "Ivan Tretyakov" <it...@griddynamics.com>
wrote:
> Thank you for your replies!
>
> We are using MR version 1 and my question is regarding this version.
>
> Omkar,
> are you talking about MR1 or MR2? I didn't find property to limit number
> of running jobs per queue for capacity scheduler using MR1. Did I wrong
> somewhere? What exactly option do you mean?
>
> Sandy,
> thanks, I got it. But unfortunately we are using MR1 for now.
>
>
> On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Unfortunately, this is not possible in the MR1 fair scheduler without
>> setting the jobs for individual pools. In MR2, fair scheduler hierarchical
>> queues will allow setting maxRunningApps at the top of the hierarchy, which
>> would have the effect you're looking for.
>>
>> -Sandy
>>
>>
>> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>>
>>> Not sure about the fair scheduler but in capacity scheduler you can
>>> achieve this by controlling number of jobs/applications per queue.
>>>
>>> Thanks,
>>> Omkar Joshi
>>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>>
>>>
>>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>>> itretyakov@griddynamics.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>>> We need to limit total number of jobs which can run at the same time on
>>>> cluster.
>>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>>> We wouldn't like to limit each pool or user we just need to set limit
>>>> on total number of jobs running.
>>>>
>>>> Is it possible to do it using fair scheduler?
>>>> Can capacity scheduler help here?
>>>> Maybe there are other options to achieve the goal?
>>>>
>>>> Thanks in advance!
>>>>
>>>> --
>>>> Best Regards
>>>> Ivan Tretyakov
>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>
Re: Limit on total jobs running using fair scheduler
Posted by Azuryy Yu <az...@gmail.com>.
fot MRv1, Its impossible.
On 2013-11-22 5:46 AM, "Ivan Tretyakov" <it...@griddynamics.com>
wrote:
> Thank you for your replies!
>
> We are using MR version 1 and my question is regarding this version.
>
> Omkar,
> are you talking about MR1 or MR2? I didn't find property to limit number
> of running jobs per queue for capacity scheduler using MR1. Did I wrong
> somewhere? What exactly option do you mean?
>
> Sandy,
> thanks, I got it. But unfortunately we are using MR1 for now.
>
>
> On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Unfortunately, this is not possible in the MR1 fair scheduler without
>> setting the jobs for individual pools. In MR2, fair scheduler hierarchical
>> queues will allow setting maxRunningApps at the top of the hierarchy, which
>> would have the effect you're looking for.
>>
>> -Sandy
>>
>>
>> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>>
>>> Not sure about the fair scheduler but in capacity scheduler you can
>>> achieve this by controlling number of jobs/applications per queue.
>>>
>>> Thanks,
>>> Omkar Joshi
>>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>>
>>>
>>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>>> itretyakov@griddynamics.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>>> We need to limit total number of jobs which can run at the same time on
>>>> cluster.
>>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>>> We wouldn't like to limit each pool or user we just need to set limit
>>>> on total number of jobs running.
>>>>
>>>> Is it possible to do it using fair scheduler?
>>>> Can capacity scheduler help here?
>>>> Maybe there are other options to achieve the goal?
>>>>
>>>> Thanks in advance!
>>>>
>>>> --
>>>> Best Regards
>>>> Ivan Tretyakov
>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>
Re: Limit on total jobs running using fair scheduler
Posted by Omkar Joshi <oj...@hortonworks.com>.
MRv2 :)
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Thu, Nov 21, 2013 at 1:46 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:
> Thank you for your replies!
>
> We are using MR version 1 and my question is regarding this version.
>
> Omkar,
> are you talking about MR1 or MR2? I didn't find property to limit number
> of running jobs per queue for capacity scheduler using MR1. Did I wrong
> somewhere? What exactly option do you mean?
>
> Sandy,
> thanks, I got it. But unfortunately we are using MR1 for now.
>
>
> On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Unfortunately, this is not possible in the MR1 fair scheduler without
>> setting the jobs for individual pools. In MR2, fair scheduler hierarchical
>> queues will allow setting maxRunningApps at the top of the hierarchy, which
>> would have the effect you're looking for.
>>
>> -Sandy
>>
>>
>> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>>
>>> Not sure about the fair scheduler but in capacity scheduler you can
>>> achieve this by controlling number of jobs/applications per queue.
>>>
>>> Thanks,
>>> Omkar Joshi
>>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>>
>>>
>>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>>> itretyakov@griddynamics.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>>> We need to limit total number of jobs which can run at the same time on
>>>> cluster.
>>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>>> We wouldn't like to limit each pool or user we just need to set limit
>>>> on total number of jobs running.
>>>>
>>>> Is it possible to do it using fair scheduler?
>>>> Can capacity scheduler help here?
>>>> Maybe there are other options to achieve the goal?
>>>>
>>>> Thanks in advance!
>>>>
>>>> --
>>>> Best Regards
>>>> Ivan Tretyakov
>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Limit on total jobs running using fair scheduler
Posted by Azuryy Yu <az...@gmail.com>.
fot MRv1, Its impossible.
On 2013-11-22 5:46 AM, "Ivan Tretyakov" <it...@griddynamics.com>
wrote:
> Thank you for your replies!
>
> We are using MR version 1 and my question is regarding this version.
>
> Omkar,
> are you talking about MR1 or MR2? I didn't find property to limit number
> of running jobs per queue for capacity scheduler using MR1. Did I wrong
> somewhere? What exactly option do you mean?
>
> Sandy,
> thanks, I got it. But unfortunately we are using MR1 for now.
>
>
> On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Unfortunately, this is not possible in the MR1 fair scheduler without
>> setting the jobs for individual pools. In MR2, fair scheduler hierarchical
>> queues will allow setting maxRunningApps at the top of the hierarchy, which
>> would have the effect you're looking for.
>>
>> -Sandy
>>
>>
>> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>>
>>> Not sure about the fair scheduler but in capacity scheduler you can
>>> achieve this by controlling number of jobs/applications per queue.
>>>
>>> Thanks,
>>> Omkar Joshi
>>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>>
>>>
>>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>>> itretyakov@griddynamics.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>>> We need to limit total number of jobs which can run at the same time on
>>>> cluster.
>>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>>> We wouldn't like to limit each pool or user we just need to set limit
>>>> on total number of jobs running.
>>>>
>>>> Is it possible to do it using fair scheduler?
>>>> Can capacity scheduler help here?
>>>> Maybe there are other options to achieve the goal?
>>>>
>>>> Thanks in advance!
>>>>
>>>> --
>>>> Best Regards
>>>> Ivan Tretyakov
>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>
Re: Limit on total jobs running using fair scheduler
Posted by Omkar Joshi <oj...@hortonworks.com>.
MRv2 :)
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Thu, Nov 21, 2013 at 1:46 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:
> Thank you for your replies!
>
> We are using MR version 1 and my question is regarding this version.
>
> Omkar,
> are you talking about MR1 or MR2? I didn't find property to limit number
> of running jobs per queue for capacity scheduler using MR1. Did I wrong
> somewhere? What exactly option do you mean?
>
> Sandy,
> thanks, I got it. But unfortunately we are using MR1 for now.
>
>
> On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Unfortunately, this is not possible in the MR1 fair scheduler without
>> setting the jobs for individual pools. In MR2, fair scheduler hierarchical
>> queues will allow setting maxRunningApps at the top of the hierarchy, which
>> would have the effect you're looking for.
>>
>> -Sandy
>>
>>
>> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>>
>>> Not sure about the fair scheduler but in capacity scheduler you can
>>> achieve this by controlling number of jobs/applications per queue.
>>>
>>> Thanks,
>>> Omkar Joshi
>>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>>
>>>
>>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>>> itretyakov@griddynamics.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>>> We need to limit total number of jobs which can run at the same time on
>>>> cluster.
>>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>>> We wouldn't like to limit each pool or user we just need to set limit
>>>> on total number of jobs running.
>>>>
>>>> Is it possible to do it using fair scheduler?
>>>> Can capacity scheduler help here?
>>>> Maybe there are other options to achieve the goal?
>>>>
>>>> Thanks in advance!
>>>>
>>>> --
>>>> Best Regards
>>>> Ivan Tretyakov
>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Limit on total jobs running using fair scheduler
Posted by Omkar Joshi <oj...@hortonworks.com>.
MRv2 :)
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Thu, Nov 21, 2013 at 1:46 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:
> Thank you for your replies!
>
> We are using MR version 1 and my question is regarding this version.
>
> Omkar,
> are you talking about MR1 or MR2? I didn't find property to limit number
> of running jobs per queue for capacity scheduler using MR1. Did I wrong
> somewhere? What exactly option do you mean?
>
> Sandy,
> thanks, I got it. But unfortunately we are using MR1 for now.
>
>
> On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Unfortunately, this is not possible in the MR1 fair scheduler without
>> setting the jobs for individual pools. In MR2, fair scheduler hierarchical
>> queues will allow setting maxRunningApps at the top of the hierarchy, which
>> would have the effect you're looking for.
>>
>> -Sandy
>>
>>
>> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>>
>>> Not sure about the fair scheduler but in capacity scheduler you can
>>> achieve this by controlling number of jobs/applications per queue.
>>>
>>> Thanks,
>>> Omkar Joshi
>>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>>
>>>
>>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>>> itretyakov@griddynamics.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>>> We need to limit total number of jobs which can run at the same time on
>>>> cluster.
>>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>>> We wouldn't like to limit each pool or user we just need to set limit
>>>> on total number of jobs running.
>>>>
>>>> Is it possible to do it using fair scheduler?
>>>> Can capacity scheduler help here?
>>>> Maybe there are other options to achieve the goal?
>>>>
>>>> Thanks in advance!
>>>>
>>>> --
>>>> Best Regards
>>>> Ivan Tretyakov
>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Limit on total jobs running using fair scheduler
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Thank you for your replies!
We are using MR version 1 and my question is regarding this version.
Omkar,
are you talking about MR1 or MR2? I didn't find property to limit number of
running jobs per queue for capacity scheduler using MR1. Did I wrong
somewhere? What exactly option do you mean?
Sandy,
thanks, I got it. But unfortunately we are using MR1 for now.
On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> Unfortunately, this is not possible in the MR1 fair scheduler without
> setting the jobs for individual pools. In MR2, fair scheduler hierarchical
> queues will allow setting maxRunningApps at the top of the hierarchy, which
> would have the effect you're looking for.
>
> -Sandy
>
>
> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>
>> Not sure about the fair scheduler but in capacity scheduler you can
>> achieve this by controlling number of jobs/applications per queue.
>>
>> Thanks,
>> Omkar Joshi
>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>
>>
>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>> We need to limit total number of jobs which can run at the same time on
>>> cluster.
>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>> We wouldn't like to limit each pool or user we just need to set limit on
>>> total number of jobs running.
>>>
>>> Is it possible to do it using fair scheduler?
>>> Can capacity scheduler help here?
>>> Maybe there are other options to achieve the goal?
>>>
>>> Thanks in advance!
>>>
>>> --
>>> Best Regards
>>> Ivan Tretyakov
>>>
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Limit on total jobs running using fair scheduler
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Thank you for your replies!
We are using MR version 1 and my question is regarding this version.
Omkar,
are you talking about MR1 or MR2? I didn't find property to limit number of
running jobs per queue for capacity scheduler using MR1. Did I wrong
somewhere? What exactly option do you mean?
Sandy,
thanks, I got it. But unfortunately we are using MR1 for now.
On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> Unfortunately, this is not possible in the MR1 fair scheduler without
> setting the jobs for individual pools. In MR2, fair scheduler hierarchical
> queues will allow setting maxRunningApps at the top of the hierarchy, which
> would have the effect you're looking for.
>
> -Sandy
>
>
> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>
>> Not sure about the fair scheduler but in capacity scheduler you can
>> achieve this by controlling number of jobs/applications per queue.
>>
>> Thanks,
>> Omkar Joshi
>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>
>>
>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>> We need to limit total number of jobs which can run at the same time on
>>> cluster.
>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>> We wouldn't like to limit each pool or user we just need to set limit on
>>> total number of jobs running.
>>>
>>> Is it possible to do it using fair scheduler?
>>> Can capacity scheduler help here?
>>> Maybe there are other options to achieve the goal?
>>>
>>> Thanks in advance!
>>>
>>> --
>>> Best Regards
>>> Ivan Tretyakov
>>>
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Limit on total jobs running using fair scheduler
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Thank you for your replies!
We are using MR version 1 and my question is regarding this version.
Omkar,
are you talking about MR1 or MR2? I didn't find property to limit number of
running jobs per queue for capacity scheduler using MR1. Did I wrong
somewhere? What exactly option do you mean?
Sandy,
thanks, I got it. But unfortunately we are using MR1 for now.
On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> Unfortunately, this is not possible in the MR1 fair scheduler without
> setting the jobs for individual pools. In MR2, fair scheduler hierarchical
> queues will allow setting maxRunningApps at the top of the hierarchy, which
> would have the effect you're looking for.
>
> -Sandy
>
>
> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>
>> Not sure about the fair scheduler but in capacity scheduler you can
>> achieve this by controlling number of jobs/applications per queue.
>>
>> Thanks,
>> Omkar Joshi
>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>
>>
>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>> We need to limit total number of jobs which can run at the same time on
>>> cluster.
>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>> We wouldn't like to limit each pool or user we just need to set limit on
>>> total number of jobs running.
>>>
>>> Is it possible to do it using fair scheduler?
>>> Can capacity scheduler help here?
>>> Maybe there are other options to achieve the goal?
>>>
>>> Thanks in advance!
>>>
>>> --
>>> Best Regards
>>> Ivan Tretyakov
>>>
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Limit on total jobs running using fair scheduler
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Thank you for your replies!
We are using MR version 1 and my question is regarding this version.
Omkar,
are you talking about MR1 or MR2? I didn't find property to limit number of
running jobs per queue for capacity scheduler using MR1. Did I wrong
somewhere? What exactly option do you mean?
Sandy,
thanks, I got it. But unfortunately we are using MR1 for now.
On Wed, Nov 20, 2013 at 2:12 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> Unfortunately, this is not possible in the MR1 fair scheduler without
> setting the jobs for individual pools. In MR2, fair scheduler hierarchical
> queues will allow setting maxRunningApps at the top of the hierarchy, which
> would have the effect you're looking for.
>
> -Sandy
>
>
> On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>
>> Not sure about the fair scheduler but in capacity scheduler you can
>> achieve this by controlling number of jobs/applications per queue.
>>
>> Thanks,
>> Omkar Joshi
>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>
>>
>> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>>> We need to limit total number of jobs which can run at the same time on
>>> cluster.
>>> I can see maxRunningJobs options but it sets limit for pool or user.
>>> We wouldn't like to limit each pool or user we just need to set limit on
>>> total number of jobs running.
>>>
>>> Is it possible to do it using fair scheduler?
>>> Can capacity scheduler help here?
>>> Maybe there are other options to achieve the goal?
>>>
>>> Thanks in advance!
>>>
>>> --
>>> Best Regards
>>> Ivan Tretyakov
>>>
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Limit on total jobs running using fair scheduler
Posted by Sandy Ryza <sa...@cloudera.com>.
Unfortunately, this is not possible in the MR1 fair scheduler without
setting the jobs for individual pools. In MR2, fair scheduler hierarchical
queues will allow setting maxRunningApps at the top of the hierarchy, which
would have the effect you're looking for.
-Sandy
On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
> Not sure about the fair scheduler but in capacity scheduler you can
> achieve this by controlling number of jobs/applications per queue.
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>> We need to limit total number of jobs which can run at the same time on
>> cluster.
>> I can see maxRunningJobs options but it sets limit for pool or user.
>> We wouldn't like to limit each pool or user we just need to set limit on
>> total number of jobs running.
>>
>> Is it possible to do it using fair scheduler?
>> Can capacity scheduler help here?
>> Maybe there are other options to achieve the goal?
>>
>> Thanks in advance!
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Limit on total jobs running using fair scheduler
Posted by Sandy Ryza <sa...@cloudera.com>.
Unfortunately, this is not possible in the MR1 fair scheduler without
setting the jobs for individual pools. In MR2, fair scheduler hierarchical
queues will allow setting maxRunningApps at the top of the hierarchy, which
would have the effect you're looking for.
-Sandy
On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
> Not sure about the fair scheduler but in capacity scheduler you can
> achieve this by controlling number of jobs/applications per queue.
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>> We need to limit total number of jobs which can run at the same time on
>> cluster.
>> I can see maxRunningJobs options but it sets limit for pool or user.
>> We wouldn't like to limit each pool or user we just need to set limit on
>> total number of jobs running.
>>
>> Is it possible to do it using fair scheduler?
>> Can capacity scheduler help here?
>> Maybe there are other options to achieve the goal?
>>
>> Thanks in advance!
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Limit on total jobs running using fair scheduler
Posted by Sandy Ryza <sa...@cloudera.com>.
Unfortunately, this is not possible in the MR1 fair scheduler without
setting the jobs for individual pools. In MR2, fair scheduler hierarchical
queues will allow setting maxRunningApps at the top of the hierarchy, which
would have the effect you're looking for.
-Sandy
On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
> Not sure about the fair scheduler but in capacity scheduler you can
> achieve this by controlling number of jobs/applications per queue.
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>> We need to limit total number of jobs which can run at the same time on
>> cluster.
>> I can see maxRunningJobs options but it sets limit for pool or user.
>> We wouldn't like to limit each pool or user we just need to set limit on
>> total number of jobs running.
>>
>> Is it possible to do it using fair scheduler?
>> Can capacity scheduler help here?
>> Maybe there are other options to achieve the goal?
>>
>> Thanks in advance!
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Limit on total jobs running using fair scheduler
Posted by Sandy Ryza <sa...@cloudera.com>.
Unfortunately, this is not possible in the MR1 fair scheduler without
setting the jobs for individual pools. In MR2, fair scheduler hierarchical
queues will allow setting maxRunningApps at the top of the hierarchy, which
would have the effect you're looking for.
-Sandy
On Tue, Nov 19, 2013 at 2:01 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
> Not sure about the fair scheduler but in capacity scheduler you can
> achieve this by controlling number of jobs/applications per queue.
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
>> We need to limit total number of jobs which can run at the same time on
>> cluster.
>> I can see maxRunningJobs options but it sets limit for pool or user.
>> We wouldn't like to limit each pool or user we just need to set limit on
>> total number of jobs running.
>>
>> Is it possible to do it using fair scheduler?
>> Can capacity scheduler help here?
>> Maybe there are other options to achieve the goal?
>>
>> Thanks in advance!
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Limit on total jobs running using fair scheduler
Posted by Omkar Joshi <oj...@hortonworks.com>.
Not sure about the fair scheduler but in capacity scheduler you can achieve
this by controlling number of jobs/applications per queue.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:
> Hello!
>
> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
> We need to limit total number of jobs which can run at the same time on
> cluster.
> I can see maxRunningJobs options but it sets limit for pool or user.
> We wouldn't like to limit each pool or user we just need to set limit on
> total number of jobs running.
>
> Is it possible to do it using fair scheduler?
> Can capacity scheduler help here?
> Maybe there are other options to achieve the goal?
>
> Thanks in advance!
>
> --
> Best Regards
> Ivan Tretyakov
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Limit on total jobs running using fair scheduler
Posted by Omkar Joshi <oj...@hortonworks.com>.
Not sure about the fair scheduler but in capacity scheduler you can achieve
this by controlling number of jobs/applications per queue.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:
> Hello!
>
> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
> We need to limit total number of jobs which can run at the same time on
> cluster.
> I can see maxRunningJobs options but it sets limit for pool or user.
> We wouldn't like to limit each pool or user we just need to set limit on
> total number of jobs running.
>
> Is it possible to do it using fair scheduler?
> Can capacity scheduler help here?
> Maybe there are other options to achieve the goal?
>
> Thanks in advance!
>
> --
> Best Regards
> Ivan Tretyakov
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Limit on total jobs running using fair scheduler
Posted by Omkar Joshi <oj...@hortonworks.com>.
Not sure about the fair scheduler but in capacity scheduler you can achieve
this by controlling number of jobs/applications per queue.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:
> Hello!
>
> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
> We need to limit total number of jobs which can run at the same time on
> cluster.
> I can see maxRunningJobs options but it sets limit for pool or user.
> We wouldn't like to limit each pool or user we just need to set limit on
> total number of jobs running.
>
> Is it possible to do it using fair scheduler?
> Can capacity scheduler help here?
> Maybe there are other options to achieve the goal?
>
> Thanks in advance!
>
> --
> Best Regards
> Ivan Tretyakov
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Limit on total jobs running using fair scheduler
Posted by Omkar Joshi <oj...@hortonworks.com>.
Not sure about the fair scheduler but in capacity scheduler you can achieve
this by controlling number of jobs/applications per queue.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Tue, Nov 19, 2013 at 3:26 AM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:
> Hello!
>
> We are using CDH 4.1.1 (Version: 2.0.0-mr1-cdh4.1.1) and fair-scheduler.
> We need to limit total number of jobs which can run at the same time on
> cluster.
> I can see maxRunningJobs options but it sets limit for pool or user.
> We wouldn't like to limit each pool or user we just need to set limit on
> total number of jobs running.
>
> Is it possible to do it using fair scheduler?
> Can capacity scheduler help here?
> Maybe there are other options to achieve the goal?
>
> Thanks in advance!
>
> --
> Best Regards
> Ivan Tretyakov
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.