Hadoop configuration defaults

[Marco Didonna <m....@gmail.com>]

Hello everyone,
I was sifting through the whirr source code in order to understand
where the default parameters in the hadoop configuration came from. In
particular I was quite a lot pissed by the ulimit parameter which was
the cause of my jvms being killed. I see there's a
whirr-hadoop-default.properties in
whirr/services/hadoop/src/main/resources, my question is: if I edit
this file will whirr obey to the new settings? Or should I perform
some kind of compilation step?

Thanks,

MD

Re: Hadoop configuration defaults

[Marco Didonna <m....@gmail.com>]

On 17 December 2011 15:51, Marco Didonna <m....@gmail.com> wrote:
> I want that if I do not specify a property
> whirr will not add it (like mapred.child.ulimit) so that it defaults
> to hadoop of that particular property.

Sorry this came out wrong, let me rephrase it: I want that whirr does
not add any properties on his own. Or at least I want to decide which
properties is it allow to add, if not overridden by the property file
I give it using --config option.

MD

Re: Hadoop configuration defaults

[Marco Didonna <m....@gmail.com>]

On 17 December 2011 15:23, Frank Scholten <fr...@frankscholten.nl> wrote:
> Hi Marco,
>
> You can add hadoop properties in your whirr properties file like this:
>
> hadoop-mapreduce.mapred.child.java.opts=-Xmx1000m
> hadoop-mapreduce.mapred.child.ulimit=1500000
>
> See http://www.searchworkings.org/blog/-/blogs/running-mahout-in-the-cloud-using-apache-whirr
>
> Cheers,
>
> Frank

Maybe I didn't make myself clear enough: I do use a configuration file
and I do add my own properties like
hadoop-mapreduce.mapred.child.java.opts=-Xmx1g. What I didn't know is
that whirr automatically and quite secretly adds its own properties
like mapred.child.ulimit. I want that if I do not specify a property
whirr will not add it (like mapred.child.ulimit) so that it defaults
to hadoop of that particular property.
I did try to modify whirr-hadoop-default.properties removing two lines
but now whirr doesn't work properly anymore: in particular it doesn't
install cdh any more.
What modification to the source code should I perform to reach my goal?

Thank you

Re: Hadoop configuration defaults

[Frank Scholten <fr...@frankscholten.nl>]

Hi Marco,

You can add hadoop properties in your whirr properties file like this:

hadoop-mapreduce.mapred.child.java.opts=-Xmx1000m
hadoop-mapreduce.mapred.child.ulimit=1500000

See http://www.searchworkings.org/blog/-/blogs/running-mahout-in-the-cloud-using-apache-whirr

Cheers,

Frank

On Sat, Dec 17, 2011 at 11:06 AM, Marco Didonna <m....@gmail.com> wrote:
> Hello everyone,
> I was sifting through the whirr source code in order to understand
> where the default parameters in the hadoop configuration came from. In
> particular I was quite a lot pissed by the ulimit parameter which was
> the cause of my jvms being killed. I see there's a
> whirr-hadoop-default.properties in
> whirr/services/hadoop/src/main/resources, my question is: if I edit
> this file will whirr obey to the new settings? Or should I perform
> some kind of compilation step?
>
> Thanks,
>
> MD