is there any way to set the niceness of map/reduce jobs

[Jinsong Hu <ji...@hotmail.com>]

Hi: there:
  I have a cluster that is used for both hadoop mapreduce and hbase. What I 
found is that when I am running map/reduce jobs, the job can be very 
memory/cpu intensive, and cause hbase or data nodes to freeze. in hbase's 
case, the region server may shut it self down.
  In order to avoid this, I made very conservative configuration of the 
maximum number of mappers and reducers. However, I am wonder if hadoop 
allows me to start map/reduce with the command "nice" so that
those jobs get lower priority than datanode/tasktracker/hbase regionserver. 
That way, if there is enough resource, the jobs can fully utilize them. but 
if not, those jobs will yield to other processes.

Jimmy. 

Re: is there any way to set the niceness of map/reduce jobs

[Brian Bockelman <bb...@cse.unl.edu>]

Not to mention on new kernels, cgroups and the new I/O scheduler should allow you to very precisely partition up I/O bandwidth among arbitrary sets of processes.  I believe Solaris Containers (and IBM's whatever-they-call-it) have had these features for years.

Candidly, I believe VMs are the incorrect solution to separate the services if performance is the issue (they may be the solution if the issue is migration, future growth, complicated deployment, or somewhat security).  You're just adding another layer that obfuscates what's happening on the hardware.

Brian

On Nov 3, 2010, at 6:11 AM, Steve Loughran wrote:

> On 03/11/10 02:05, Lance Norskog wrote:
>> Virtual Machines like VMWare give you a lot of control over this.
>> They're a pain to set up but in production this might be the most
>> stable solution.
>> 
> gives you a big IO perf hit though. VMs are good for agile clusters which only exist for a short period of time -and whose size varies on load. You can also set up a secure cluster. But for turning perf problems, I'm not sure its the right answer
> 
> 
> -as well as ionice, there is iotop to measure what is doing the IO load. You need to know what the problem is (CPU, IO, garbage collection, contention for the namenode, swapping) to decide what to tune.
> 
> Anything anyone could write up on this would be lovely.
> 
> -steve

Re: is there any way to set the niceness of map/reduce jobs

[Steve Loughran <st...@apache.org>]

On 03/11/10 02:05, Lance Norskog wrote:
> Virtual Machines like VMWare give you a lot of control over this.
> They're a pain to set up but in production this might be the most
> stable solution.
>
gives you a big IO perf hit though. VMs are good for agile clusters 
which only exist for a short period of time -and whose size varies on 
load. You can also set up a secure cluster. But for turning perf 
problems, I'm not sure its the right answer


-as well as ionice, there is iotop to measure what is doing the IO load. 
You need to know what the problem is (CPU, IO, garbage collection, 
contention for the namenode, swapping) to decide what to tune.

Anything anyone could write up on this would be lovely.

-steve

Re: is there any way to set the niceness of map/reduce jobs

[Lance Norskog <go...@gmail.com>]

Virtual Machines like VMWare give you a lot of control over this.
They're a pain to set up but in production this might be the most
stable solution.


On Tue, Nov 2, 2010 at 6:17 PM, Luke Lu <ll...@vicaya.com> wrote:
> nice(1) only changes cpu scheduling priority. It doesn't really help
> if you have tasks (and their child processes) that use too much
> memory, which causes swapping, which is probably the real culprit to
> cause servers to freeze. Decreasing kernel swappiness probably helps.
> Another thing to try is ionice (on linux if you have reasonably recent
> kernel with cfq as io scheduler, default for rhel5) if the freeze is
> caused by io contention (assuming no swapping.)
>
> You can write a simple script to periodically renice(1) and ionice(1)
> these processes to see if they actually work for you.
>
> On Tue, Nov 2, 2010 at 4:51 PM, Jinsong Hu <ji...@hotmail.com> wrote:
>> Hi: there:
>>  I have a cluster that is used for both hadoop mapreduce and hbase. What I
>> found is that when I am running map/reduce jobs, the job can be very
>> memory/cpu intensive, and cause hbase or data nodes to freeze. in hbase's
>> case, the region server may shut it self down.
>>  In order to avoid this, I made very conservative configuration of the
>> maximum number of mappers and reducers. However, I am wonder if hadoop
>> allows me to start map/reduce with the command "nice" so that
>> those jobs get lower priority than datanode/tasktracker/hbase regionserver.
>> That way, if there is enough resource, the jobs can fully utilize them. but
>> if not, those jobs will yield to other processes.
>>
>> Jimmy.
>>
>



-- 
Lance Norskog
goksron@gmail.com

Re: is there any way to set the niceness of map/reduce jobs

[Luke Lu <ll...@vicaya.com>]

nice(1) only changes cpu scheduling priority. It doesn't really help
if you have tasks (and their child processes) that use too much
memory, which causes swapping, which is probably the real culprit to
cause servers to freeze. Decreasing kernel swappiness probably helps.
Another thing to try is ionice (on linux if you have reasonably recent
kernel with cfq as io scheduler, default for rhel5) if the freeze is
caused by io contention (assuming no swapping.)

You can write a simple script to periodically renice(1) and ionice(1)
these processes to see if they actually work for you.

On Tue, Nov 2, 2010 at 4:51 PM, Jinsong Hu <ji...@hotmail.com> wrote:
> Hi: there:
>  I have a cluster that is used for both hadoop mapreduce and hbase. What I
> found is that when I am running map/reduce jobs, the job can be very
> memory/cpu intensive, and cause hbase or data nodes to freeze. in hbase's
> case, the region server may shut it self down.
>  In order to avoid this, I made very conservative configuration of the
> maximum number of mappers and reducers. However, I am wonder if hadoop
> allows me to start map/reduce with the command "nice" so that
> those jobs get lower priority than datanode/tasktracker/hbase regionserver.
> That way, if there is enough resource, the jobs can fully utilize them. but
> if not, those jobs will yield to other processes.
>
> Jimmy.
>