You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jay Vyas <ja...@gmail.com> on 2012/05/09 22:09:09 UTC
Simulating cluster config parameters (in local mode)
Hi guys : I need to set a cluster configuration parameter (specifically,
the "mapreduce.job.counters.limit") ....
.... Easy ... right ?
Well.... one problem : I'm running hadoop in local-mode !
So How can I simulate this parameter so that my local mode allows me to use
non-default cluster configruation parameters ?
--
Jay Vyas
MMSB/UCHC
Re: Simulating cluster config parameters (in local mode)
Posted by Harsh J <ha...@cloudera.com>.
Jay,
Yep even the local mode does read mapred-site.xml, and a few
properties supplied it in apply to LocalJobRunner as well.
Yes the per-job override will not work for raising limits, as that
wouldn't make sense as a 'limit' then (Although Android has a
different sense of that I've noticed:
http://developer.android.com/reference/android/os/StrictMode.html,
which works for its devs and is a different playground :))
On Thu, May 10, 2012 at 2:44 AM, Jay Vyas <ja...@gmail.com> wrote:
> Ahhh I now know the answer : My solution :
>
> 1) Get a simple mapred config file and remove all parameters but the one i
> need to set for my local mode.
> 2) Put the mapred site config .xml in my classpath.
> 3) Run my application.
>
> The JobConf I assume SHOULD NOT work because this parameter is specifically
> meant to defy and job specific configuration.
>
>
> On Wed, May 9, 2012 at 5:03 PM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com> wrote:
>
>> You should be able to set that param on JobConf object
>>
>> Regards,
>> Serge
>>
>> On 5/9/12 1:09 PM, "Jay Vyas" <ja...@gmail.com> wrote:
>>
>> >Hi guys : I need to set a cluster configuration parameter (specifically,
>> >the "mapreduce.job.counters.limit") ....
>> >.... Easy ... right ?
>> >
>> >Well.... one problem : I'm running hadoop in local-mode !
>> >
>> >So How can I simulate this parameter so that my local mode allows me to
>> >use
>> >non-default cluster configruation parameters ?
>> >
>> >
>> >--
>> >Jay Vyas
>> >MMSB/UCHC
>>
>>
>
>
> --
> Jay Vyas
> MMSB/UCHC
--
Harsh J
Re: Simulating cluster config parameters (in local mode)
Posted by Jay Vyas <ja...@gmail.com>.
Ahhh I now know the answer : My solution :
1) Get a simple mapred config file and remove all parameters but the one i
need to set for my local mode.
2) Put the mapred site config .xml in my classpath.
3) Run my application.
The JobConf I assume SHOULD NOT work because this parameter is specifically
meant to defy and job specific configuration.
On Wed, May 9, 2012 at 5:03 PM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> You should be able to set that param on JobConf object
>
> Regards,
> Serge
>
> On 5/9/12 1:09 PM, "Jay Vyas" <ja...@gmail.com> wrote:
>
> >Hi guys : I need to set a cluster configuration parameter (specifically,
> >the "mapreduce.job.counters.limit") ....
> >.... Easy ... right ?
> >
> >Well.... one problem : I'm running hadoop in local-mode !
> >
> >So How can I simulate this parameter so that my local mode allows me to
> >use
> >non-default cluster configruation parameters ?
> >
> >
> >--
> >Jay Vyas
> >MMSB/UCHC
>
>
--
Jay Vyas
MMSB/UCHC
Re: Simulating cluster config parameters (in local mode)
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
You should be able to set that param on JobConf object
Regards,
Serge
On 5/9/12 1:09 PM, "Jay Vyas" <ja...@gmail.com> wrote:
>Hi guys : I need to set a cluster configuration parameter (specifically,
>the "mapreduce.job.counters.limit") ....
>.... Easy ... right ?
>
>Well.... one problem : I'm running hadoop in local-mode !
>
>So How can I simulate this parameter so that my local mode allows me to
>use
>non-default cluster configruation parameters ?
>
>
>--
>Jay Vyas
>MMSB/UCHC