concurrent job execution

[Felix Sprick <fs...@gmail.com>]

Hi,

We are running MapReduce on Hbase tables and are trying to implement a
scenario with MapReduce where tasks are submitted from a GUI application.
This means that several users (currently 5-10) may use the system in
parallel. We noticed that the jobs are executed sequentially, which doesnt
really fit our use-cases. I was wondering if there are any configuration
changes we can do to the system to allow several jobs to be executed
concurrently. Our setup has 1 master and 4 slaves with each slave running a
TaskTracker and RegionServer.

Any help or hints are much appreciated.

thanks,
Felix

Re: concurrent job execution

[Arun C Murthy <ac...@yahoo-inc.com>]

Looks like you are using the default (fifo) scheduler.

Other schedulers (CapacityScheduler/FifoScheduler) have better support  
for multiple users/queues etc. You might want to try them.

Some more info on CapacityScheduler: http://developer.yahoo.com/blogs/hadoop/posts/2011/02/capacity-scheduler/

Arun

On Jun 3, 2011, at 1:41 PM, Felix Sprick wrote:

> Hi,
>
> We are running MapReduce on Hbase tables and are trying to implement  
> a scenario with MapReduce where tasks are submitted from a GUI  
> application. This means that several users (currently 5-10) may use  
> the system in parallel. We noticed that the jobs are executed  
> sequentially, which doesnt really fit our use-cases. I was wondering  
> if there are any configuration changes we can do to the system to  
> allow several jobs to be executed concurrently. Our setup has 1  
> master and 4 slaves with each slave running a TaskTracker and  
> RegionServer.
>
> Any help or hints are much appreciated.
>
> thanks,
> Felix

Re: concurrent job execution

[Allen Wittenauer <aw...@apache.org>]

On Jun 3, 2011, at 1:11 AM, Felix Sprick wrote:

> Hi,
> 
> We are running MapReduce on Hbase tables and are trying to implement a
> scenario with MapReduce where tasks are submitted from a GUI application.
> This means that several users (currently 5-10) may use the system in
> parallel. 

	In addition to moving from the FIFO/default scheduler, make sure your GUI application actually submits the jobs as different users.   If they are the same user, most (all?) schedulers will still FIFO the jobs.

	 (-Dhadoop.job.ugi in 0.20.2 and lower, doAs() in 0.20.203 and higher).