You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Amit Sela <am...@infolinks.com> on 2013/07/08 12:53:39 UTC
Fair Scheduler pools regardless of users
Hi all,
I was wondering if there is a way to let fair scheduler ignore the user and
submit a job to a specific pool.
I would like to have 3/4 pools:
1. Very short (~1 min) routine jobs.
2. Normal processing time (<1 hr) routine jobs.
3. Long (days) experimental jobs.
4. ? ad hoc immediate jobs ?
Users matter less in my use case.
Thanks,
Amit.
(Running 20 nodes cluster with Hadoop 1.0.4)
Re: Fair Scheduler pools regardless of users
Posted by Patai Sangbutsarakum <si...@gmail.com>.
'-Dmapred.fairscheduler.pool=pool_name' will just shoot to the specified
pool.
/server/hadoop/bin/hadoop jar hadoop-examples-0.20.2-cdh3u0.jar wordcount
-Dmapred.fairscheduler.pool=pool_name input output
On Mon, Jul 8, 2013 at 3:11 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Amit,
>
> You can set the mapred.job.queue.name to the name of the queue you want
> to submit it to. If the queue does not exist, it will be created. If you
> want to configure a queue with particular properties, you can do so in the
> Fair Scheduler allocations file. See
> http://hadoop.apache.org/docs/stable/fair_scheduler.html.
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 3:53 AM, Amit Sela <am...@infolinks.com> wrote:
>
>> Hi all,
>>
>> I was wondering if there is a way to let fair scheduler ignore the user
>> and submit a job to a specific pool.
>> I would like to have 3/4 pools:
>>
>> 1. Very short (~1 min) routine jobs.
>> 2. Normal processing time (<1 hr) routine jobs.
>> 3. Long (days) experimental jobs.
>> 4. ? ad hoc immediate jobs ?
>>
>> Users matter less in my use case.
>>
>> Thanks,
>>
>> Amit.
>>
>> (Running 20 nodes cluster with Hadoop 1.0.4)
>>
>>
>>
>
>
Re: Fair Scheduler pools regardless of users
Posted by Patai Sangbutsarakum <si...@gmail.com>.
'-Dmapred.fairscheduler.pool=pool_name' will just shoot to the specified
pool.
/server/hadoop/bin/hadoop jar hadoop-examples-0.20.2-cdh3u0.jar wordcount
-Dmapred.fairscheduler.pool=pool_name input output
On Mon, Jul 8, 2013 at 3:11 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Amit,
>
> You can set the mapred.job.queue.name to the name of the queue you want
> to submit it to. If the queue does not exist, it will be created. If you
> want to configure a queue with particular properties, you can do so in the
> Fair Scheduler allocations file. See
> http://hadoop.apache.org/docs/stable/fair_scheduler.html.
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 3:53 AM, Amit Sela <am...@infolinks.com> wrote:
>
>> Hi all,
>>
>> I was wondering if there is a way to let fair scheduler ignore the user
>> and submit a job to a specific pool.
>> I would like to have 3/4 pools:
>>
>> 1. Very short (~1 min) routine jobs.
>> 2. Normal processing time (<1 hr) routine jobs.
>> 3. Long (days) experimental jobs.
>> 4. ? ad hoc immediate jobs ?
>>
>> Users matter less in my use case.
>>
>> Thanks,
>>
>> Amit.
>>
>> (Running 20 nodes cluster with Hadoop 1.0.4)
>>
>>
>>
>
>
Re: Fair Scheduler pools regardless of users
Posted by Patai Sangbutsarakum <si...@gmail.com>.
'-Dmapred.fairscheduler.pool=pool_name' will just shoot to the specified
pool.
/server/hadoop/bin/hadoop jar hadoop-examples-0.20.2-cdh3u0.jar wordcount
-Dmapred.fairscheduler.pool=pool_name input output
On Mon, Jul 8, 2013 at 3:11 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Amit,
>
> You can set the mapred.job.queue.name to the name of the queue you want
> to submit it to. If the queue does not exist, it will be created. If you
> want to configure a queue with particular properties, you can do so in the
> Fair Scheduler allocations file. See
> http://hadoop.apache.org/docs/stable/fair_scheduler.html.
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 3:53 AM, Amit Sela <am...@infolinks.com> wrote:
>
>> Hi all,
>>
>> I was wondering if there is a way to let fair scheduler ignore the user
>> and submit a job to a specific pool.
>> I would like to have 3/4 pools:
>>
>> 1. Very short (~1 min) routine jobs.
>> 2. Normal processing time (<1 hr) routine jobs.
>> 3. Long (days) experimental jobs.
>> 4. ? ad hoc immediate jobs ?
>>
>> Users matter less in my use case.
>>
>> Thanks,
>>
>> Amit.
>>
>> (Running 20 nodes cluster with Hadoop 1.0.4)
>>
>>
>>
>
>
Re: Fair Scheduler pools regardless of users
Posted by Patai Sangbutsarakum <si...@gmail.com>.
'-Dmapred.fairscheduler.pool=pool_name' will just shoot to the specified
pool.
/server/hadoop/bin/hadoop jar hadoop-examples-0.20.2-cdh3u0.jar wordcount
-Dmapred.fairscheduler.pool=pool_name input output
On Mon, Jul 8, 2013 at 3:11 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Amit,
>
> You can set the mapred.job.queue.name to the name of the queue you want
> to submit it to. If the queue does not exist, it will be created. If you
> want to configure a queue with particular properties, you can do so in the
> Fair Scheduler allocations file. See
> http://hadoop.apache.org/docs/stable/fair_scheduler.html.
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 3:53 AM, Amit Sela <am...@infolinks.com> wrote:
>
>> Hi all,
>>
>> I was wondering if there is a way to let fair scheduler ignore the user
>> and submit a job to a specific pool.
>> I would like to have 3/4 pools:
>>
>> 1. Very short (~1 min) routine jobs.
>> 2. Normal processing time (<1 hr) routine jobs.
>> 3. Long (days) experimental jobs.
>> 4. ? ad hoc immediate jobs ?
>>
>> Users matter less in my use case.
>>
>> Thanks,
>>
>> Amit.
>>
>> (Running 20 nodes cluster with Hadoop 1.0.4)
>>
>>
>>
>
>
Re: Fair Scheduler pools regardless of users
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Amit,
You can set the mapred.job.queue.name to the name of the queue you want to
submit it to. If the queue does not exist, it will be created. If you
want to configure a queue with particular properties, you can do so in the
Fair Scheduler allocations file. See
http://hadoop.apache.org/docs/stable/fair_scheduler.html.
-Sandy
On Mon, Jul 8, 2013 at 3:53 AM, Amit Sela <am...@infolinks.com> wrote:
> Hi all,
>
> I was wondering if there is a way to let fair scheduler ignore the user
> and submit a job to a specific pool.
> I would like to have 3/4 pools:
>
> 1. Very short (~1 min) routine jobs.
> 2. Normal processing time (<1 hr) routine jobs.
> 3. Long (days) experimental jobs.
> 4. ? ad hoc immediate jobs ?
>
> Users matter less in my use case.
>
> Thanks,
>
> Amit.
>
> (Running 20 nodes cluster with Hadoop 1.0.4)
>
>
>
Re: Fair Scheduler pools regardless of users
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Amit,
You can set the mapred.job.queue.name to the name of the queue you want to
submit it to. If the queue does not exist, it will be created. If you
want to configure a queue with particular properties, you can do so in the
Fair Scheduler allocations file. See
http://hadoop.apache.org/docs/stable/fair_scheduler.html.
-Sandy
On Mon, Jul 8, 2013 at 3:53 AM, Amit Sela <am...@infolinks.com> wrote:
> Hi all,
>
> I was wondering if there is a way to let fair scheduler ignore the user
> and submit a job to a specific pool.
> I would like to have 3/4 pools:
>
> 1. Very short (~1 min) routine jobs.
> 2. Normal processing time (<1 hr) routine jobs.
> 3. Long (days) experimental jobs.
> 4. ? ad hoc immediate jobs ?
>
> Users matter less in my use case.
>
> Thanks,
>
> Amit.
>
> (Running 20 nodes cluster with Hadoop 1.0.4)
>
>
>
Re: Fair Scheduler pools regardless of users
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Amit,
You can set the mapred.job.queue.name to the name of the queue you want to
submit it to. If the queue does not exist, it will be created. If you
want to configure a queue with particular properties, you can do so in the
Fair Scheduler allocations file. See
http://hadoop.apache.org/docs/stable/fair_scheduler.html.
-Sandy
On Mon, Jul 8, 2013 at 3:53 AM, Amit Sela <am...@infolinks.com> wrote:
> Hi all,
>
> I was wondering if there is a way to let fair scheduler ignore the user
> and submit a job to a specific pool.
> I would like to have 3/4 pools:
>
> 1. Very short (~1 min) routine jobs.
> 2. Normal processing time (<1 hr) routine jobs.
> 3. Long (days) experimental jobs.
> 4. ? ad hoc immediate jobs ?
>
> Users matter less in my use case.
>
> Thanks,
>
> Amit.
>
> (Running 20 nodes cluster with Hadoop 1.0.4)
>
>
>
Re: Fair Scheduler pools regardless of users
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Amit,
You can set the mapred.job.queue.name to the name of the queue you want to
submit it to. If the queue does not exist, it will be created. If you
want to configure a queue with particular properties, you can do so in the
Fair Scheduler allocations file. See
http://hadoop.apache.org/docs/stable/fair_scheduler.html.
-Sandy
On Mon, Jul 8, 2013 at 3:53 AM, Amit Sela <am...@infolinks.com> wrote:
> Hi all,
>
> I was wondering if there is a way to let fair scheduler ignore the user
> and submit a job to a specific pool.
> I would like to have 3/4 pools:
>
> 1. Very short (~1 min) routine jobs.
> 2. Normal processing time (<1 hr) routine jobs.
> 3. Long (days) experimental jobs.
> 4. ? ad hoc immediate jobs ?
>
> Users matter less in my use case.
>
> Thanks,
>
> Amit.
>
> (Running 20 nodes cluster with Hadoop 1.0.4)
>
>
>