Passing Properties With Whitespace To Streaming

[Brian Vargas <br...@ardvaark.net>]

Hi,

Using Hadoop 0.20 (CDH2) I'm trying to pass some JVM options to my child
tasks on the command-line, like this:

$ hadoop jar streaming.jar -D mapred.reduce.tasks=0 -D
'mapred.child.java.opts=-Xms200m -Xmx400m' -input foo.txt -output bar
-mapper /bin/cat

However, this fails with:
ERROR streaming.StreamJob: Unrecognized option: -Xmx400m

This used to work in 0.18 (although using -jobconf instead of -D).  Did
the generic command-line parser change between 0.18 and 0.20 to further
split the arguments on whitespace?  If so, how would one pass property
values containing whitespace?

Thanks for any thoughts!  Best!

Brian

Re: Passing Properties With Whitespace To Streaming

[Brian Vargas <br...@ardvaark.net>]

Todd,

I am using CDH2, and solution 'a' fixed the problem.

Thanks for the help, and I look forward to the next release!

Brian

Todd Lipcon wrote:
> Hi Brian,
> 
> Any chance you are using the Cloudera distribution? We did accidentally ship
> a bug like this which will be ameliorated in our next release.
> 
> The temporary workarounds are:
> 
> a) edit /usr/bin/hadoop and change the $* to a "$@"  (including the quotes!)
> or
> b) use /usr/lib/hadoop-0.20/bin/hadoop instead of /usr/bin/hadoop
> 
> The fix (a) will be included in our release due out the end of this week.
> 
> Thanks
> -Todd
> 
> On Wed, Oct 28, 2009 at 12:10 PM, Brian Vargas <br...@ardvaark.net> wrote:
> 
>> Hi,
>>
>> Using Hadoop 0.20 (CDH2) I'm trying to pass some JVM options to my child
>> tasks on the command-line, like this:
>>
>> $ hadoop jar streaming.jar -D mapred.reduce.tasks=0 -D
>> 'mapred.child.java.opts=-Xms200m -Xmx400m' -input foo.txt -output bar
>> -mapper /bin/cat
>>
>> However, this fails with:
>> ERROR streaming.StreamJob: Unrecognized option: -Xmx400m
>>
>> This used to work in 0.18 (although using -jobconf instead of -D).  Did
>> the generic command-line parser change between 0.18 and 0.20 to further
>> split the arguments on whitespace?  If so, how would one pass property
>> values containing whitespace?
>>
>> Thanks for any thoughts!  Best!
>>
>> Brian
>>
>>
> 

Re: Passing Properties With Whitespace To Streaming

[Todd Lipcon <to...@cloudera.com>]

Hi Brian,

Any chance you are using the Cloudera distribution? We did accidentally ship
a bug like this which will be ameliorated in our next release.

The temporary workarounds are:

a) edit /usr/bin/hadoop and change the $* to a "$@"  (including the quotes!)
or
b) use /usr/lib/hadoop-0.20/bin/hadoop instead of /usr/bin/hadoop

The fix (a) will be included in our release due out the end of this week.

Thanks
-Todd

On Wed, Oct 28, 2009 at 12:10 PM, Brian Vargas <br...@ardvaark.net> wrote:

> Hi,
>
> Using Hadoop 0.20 (CDH2) I'm trying to pass some JVM options to my child
> tasks on the command-line, like this:
>
> $ hadoop jar streaming.jar -D mapred.reduce.tasks=0 -D
> 'mapred.child.java.opts=-Xms200m -Xmx400m' -input foo.txt -output bar
> -mapper /bin/cat
>
> However, this fails with:
> ERROR streaming.StreamJob: Unrecognized option: -Xmx400m
>
> This used to work in 0.18 (although using -jobconf instead of -D).  Did
> the generic command-line parser change between 0.18 and 0.20 to further
> split the arguments on whitespace?  If so, how would one pass property
> values containing whitespace?
>
> Thanks for any thoughts!  Best!
>
> Brian
>
>