Any mechanism in Hadoop to run in background

[yun peng <pe...@gmail.com>]

Hi, All...
We have a user case intended to run Mapreduce in background, while the
server serves online operations. The MapReduce job may have lower priority
comparing to the online jobs..

I know this is a different use case of Mapreduce comparing to its
originally targeted scenario (where Mapreduce largely own resource
exclusively)... But I want to know if there is any tuning knobs that allow
Mapreduce to run in low priority/with limited resource.

Thanks,
Yun

Re: Any mechanism in Hadoop to run in background

[Suraj Varma <sv...@gmail.com>]

Yes, you can change your task tracker startup script to use nice and ionice
and restart the task tracker process. The mappers and reducers spun off
this task tracker will inherit the niceness.

See the first comment in
http://blog.cloudera.com/blog/2011/04/hbase-dos-and-donts/
Quoting:
change the hadoop-0.20-tasktracker so the process is started like this:

daemon *nice -n 19 ionice -c2 -n7*/usr/lib/hadoop-0.20/bin/hadoop-daemon.sh –
config “/etc/hadoop-0.20/conf” start tasktracker $DAEMON_FLAGS

--S



On Sat, Jun 22, 2013 at 7:55 AM, yun peng <pe...@gmail.com> wrote:

> Hi, All...
> We have a user case intended to run Mapreduce in background, while the
> server serves online operations. The MapReduce job may have lower priority
> comparing to the online jobs..
>
> I know this is a different use case of Mapreduce comparing to its
> originally targeted scenario (where Mapreduce largely own resource
> exclusively)... But I want to know if there is any tuning knobs that allow
> Mapreduce to run in low priority/with limited resource.
>
> Thanks,
> Yun
>