You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Brad Ruderman <br...@radiumone.com> on 2013/08/30 18:37:42 UTC
Mappers per job per user
Hi All-
I was hoping to gather some insight in how the hadoop (and or hive) job
scheduler distributes mappers per user. I am running into an issue where I
see that hadoop (and or hive) is evenly distributing mappers per user
instead of per job.
For example:
-We have 1000 mapper capacity
-10 Jobs are running total under User A
-Each job is using 100 mappers
-User B starts a job
-User B's job is allocated 250 mappers
-User A's jobs decrease to 75 mappers each instead of 100
What could be causing this allocation to occur by user and job, instead of
just by job. I reviewed the hive/hadoop documentation and wasn't able to
find any references to distributing jobs by user.
All of the jobs are being executed within the hive shell, or using the hive
command.
Thanks,
Brad
Re: Mappers per job per user
Posted by Brad Ruderman <br...@radiumone.com>.
Hi Kumar and Ravi-
Thanks for your quick responses. Although I don't believe changing anything
from default, it is possible because we are using a distribution. I will
check.
Thanks!
Brad
On Fri, Aug 30, 2013 at 1:46 PM, kumar y <yk...@gmail.com> wrote:
>
> and you can find that by looking at your mapred-site.xml . look for this
> property
>
> mapred-site.xml: <name>mapred.jobtracker.taskScheduler</name>
>
> you should be using the default FIFO or
> <value>org.apache.hadoop.mapred.FairScheduler</value> or
> <value>org.apache.hadoop.mapred.CapacityTaskScheduler</value>
>
> Depends on what scheduler you are using , please read thru the following
> link to understand how the scheduler works ( like allocation , preemption
> etc ) .
>
>
> http://hadoop.apache.org/docs/stable/capacity_scheduler.html
> http://hadoop.apache.org/docs/stable/fair_scheduler.html
>
>
>
>>
>> ---------- Forwarded message ----------
>> From: Ravi Kiran <ma...@gmail.com>
>> Date: Fri, Aug 30, 2013 at 10:30 AM
>> Subject: Re: Mappers per job per user
>> To: user@hive.apache.org
>>
>>
>> Hi Brad,
>>
>> I believe you have configured a Capacity Scheduler for scheduling the
>> jobs rather than the default FIFO scheduler.
>>
>> Regards
>> Ravi Magham
>>
>>
>> On Fri, Aug 30, 2013 at 10:07 PM, Brad Ruderman <br...@radiumone.com>wrote:
>>
>>> Hi All-
>>> I was hoping to gather some insight in how the hadoop (and or hive) job
>>> scheduler distributes mappers per user. I am running into an issue where I
>>> see that hadoop (and or hive) is evenly distributing mappers per user
>>> instead of per job.
>>>
>>> For example:
>>> -We have 1000 mapper capacity
>>> -10 Jobs are running total under User A
>>> -Each job is using 100 mappers
>>> -User B starts a job
>>> -User B's job is allocated 250 mappers
>>> -User A's jobs decrease to 75 mappers each instead of 100
>>>
>>> What could be causing this allocation to occur by user and job, instead
>>> of just by job. I reviewed the hive/hadoop documentation and wasn't able to
>>> find any references to distributing jobs by user.
>>>
>>> All of the jobs are being executed within the hive shell, or using the
>>> hive command.
>>>
>>> Thanks,
>>> Brad
>>>
>>
>>
>>
>
Re: Mappers per job per user
Posted by kumar y <yk...@gmail.com>.
and you can find that by looking at your mapred-site.xml . look for this
property
mapred-site.xml: <name>mapred.jobtracker.taskScheduler</name>
you should be using the default FIFO or
<value>org.apache.hadoop.mapred.FairScheduler</value> or
<value>org.apache.hadoop.mapred.CapacityTaskScheduler</value>
Depends on what scheduler you are using , please read thru the following
link to understand how the scheduler works ( like allocation , preemption
etc ) .
http://hadoop.apache.org/docs/stable/capacity_scheduler.html
http://hadoop.apache.org/docs/stable/fair_scheduler.html
>
> ---------- Forwarded message ----------
> From: Ravi Kiran <ma...@gmail.com>
> Date: Fri, Aug 30, 2013 at 10:30 AM
> Subject: Re: Mappers per job per user
> To: user@hive.apache.org
>
>
> Hi Brad,
>
> I believe you have configured a Capacity Scheduler for scheduling the
> jobs rather than the default FIFO scheduler.
>
> Regards
> Ravi Magham
>
>
> On Fri, Aug 30, 2013 at 10:07 PM, Brad Ruderman <br...@radiumone.com>wrote:
>
>> Hi All-
>> I was hoping to gather some insight in how the hadoop (and or hive) job
>> scheduler distributes mappers per user. I am running into an issue where I
>> see that hadoop (and or hive) is evenly distributing mappers per user
>> instead of per job.
>>
>> For example:
>> -We have 1000 mapper capacity
>> -10 Jobs are running total under User A
>> -Each job is using 100 mappers
>> -User B starts a job
>> -User B's job is allocated 250 mappers
>> -User A's jobs decrease to 75 mappers each instead of 100
>>
>> What could be causing this allocation to occur by user and job, instead
>> of just by job. I reviewed the hive/hadoop documentation and wasn't able to
>> find any references to distributing jobs by user.
>>
>> All of the jobs are being executed within the hive shell, or using the
>> hive command.
>>
>> Thanks,
>> Brad
>>
>
>
>
Re: Mappers per job per user
Posted by Ravi Kiran <ma...@gmail.com>.
Hi Brad,
I believe you have configured a Capacity Scheduler for scheduling the
jobs rather than the default FIFO scheduler.
Regards
Ravi Magham
On Fri, Aug 30, 2013 at 10:07 PM, Brad Ruderman <br...@radiumone.com>wrote:
> Hi All-
> I was hoping to gather some insight in how the hadoop (and or hive) job
> scheduler distributes mappers per user. I am running into an issue where I
> see that hadoop (and or hive) is evenly distributing mappers per user
> instead of per job.
>
> For example:
> -We have 1000 mapper capacity
> -10 Jobs are running total under User A
> -Each job is using 100 mappers
> -User B starts a job
> -User B's job is allocated 250 mappers
> -User A's jobs decrease to 75 mappers each instead of 100
>
> What could be causing this allocation to occur by user and job, instead of
> just by job. I reviewed the hive/hadoop documentation and wasn't able to
> find any references to distributing jobs by user.
>
> All of the jobs are being executed within the hive shell, or using the
> hive command.
>
> Thanks,
> Brad
>