Change Map-Red Parameters

[Adarsh Sharma <ad...@orkash.com>]

Dear all,

Is it possible in Hadoop Map-Reduce Framework to change the job 
execution parameters during runtime.

I know we can set through job.setNumTasks and many parameters but this 
remains only for that job. In Hadoop Pipes we can set it through -D 
parameter. But the problem is other.

What I want is to run a MAp-Reduce program that changes the 
mapred-site.xml parameters and the changes got reflected in the next jobs.

Please guide me if there is a way to do this.


Thanks & best Regards
Adarsh Sharma

Re: Change Map-Red Parameters

[Sreekanth Ramakrishnan <sr...@yahoo-inc.com>]

Hi Adarsh,

Configuration class supports addition of external resources other than default Hadoop MapRed resources. You can define your own resource file. For instance like mapred-site.xml you can have myjobs-site.xml. You can define your job specific parameters in the same.

In general mapred-site.xml contains common set of configurations which a job client uses to submit the given job. Changing it dynamically can potentially cause debugging pains later. So an ideal way would be defining new configuration file, adding the same to classpath and programmatically setting the same as resource while creating your job configuration. You can periodically make changes to the newly defined resource. So if you want to override some mapred specific parameters in the same, just see to that you load the newly created resources after mapred resource files are loaded.

HTH
Sreekanth



On 3/18/11 10:03 AM, "Adarsh Sharma" <ad...@orkash.com> wrote:

Dear all,

Is it possible in Hadoop Map-Reduce Framework to change the job
execution parameters during runtime.

I know we can set through job.setNumTasks and many parameters but this
remains only for that job. In Hadoop Pipes we can set it through -D
parameter. But the problem is other.

What I want is to run a MAp-Reduce program that changes the
mapred-site.xml parameters and the changes got reflected in the next jobs.

Please guide me if there is a way to do this.


Thanks & best Regards
Adarsh Sharma


--
Sreekanth Ramakrishnan