map/reduce job conf settings ignored? (bug?)

["Brendan W." <bw...@gmail.com>]

I've passed in the following line through the code for my m/r job:

conf.set("mapred.tasktracker.reduce.tasks.maximum", "4"), wanting that to
override the value of 8 set in my mapred-conf.site files on my ten-node
cluster.

Sure enough, when I look at the job configuration in the web UI for the
jobTracker, I see that value set to four, as desired.

But the number of reduce tasks currently running is 80, as before.  Should
be a maximum of 40 now, with the new value of 4 for the property above.

So what I'm wondering is, is that value of 4 being ignored somehow?  Seems
like a bug if so...

Thanks for any help...

Re: map/reduce job conf settings ignored? (bug?)

[Keith Wiley <kw...@keithwiley.com>]

Also note that the property may have been set up as "final" meaning you can't override it (without reconfiguring the cluster).  I had a problem like this.

On Mar 4, 2011, at 8:43 AM, Harsh J wrote:

> As the name goes (mapred.tasktracker.*), that is a TaskTracker
> initialization property, not a Job-specific one. Can only be changed
> in mapred-site.xml (after which the TaskTracker may need to be
> restarted to reflect the change)
> 
> -- 
> Harsh J
> www.harshj.com


________________________________________________________________________________
Keith Wiley     kwiley@keithwiley.com     keithwiley.com    music.keithwiley.com

"And what if we picked the wrong religion?  Every week, we're just making God
madder and madder!"
                                           --  Homer Simpson
________________________________________________________________________________

Re: map/reduce job conf settings ignored? (bug?)

[Harsh J <qw...@gmail.com>]

As the name goes (mapred.tasktracker.*), that is a TaskTracker
initialization property, not a Job-specific one. Can only be changed
in mapred-site.xml (after which the TaskTracker may need to be
restarted to reflect the change)

-- 
Harsh J
www.harshj.com