You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Gopal Gandhi <go...@yahoo.com> on 2008/08/21 09:14:56 UTC
[Streaming] How to pass arguments to a map/reduce script
I am using Hadoop streaming and I need to pass arguments to my map/reduce script. Because a map/reduce script is triggered by hadoop, like
hadoop .... -file MAPPER -mapper "$MAPPER" -file REDUCER -reducer "$REDUCER" ...
How can I pass arguments to MAPPER?
I tried -cmdenv name=val , but it does not work.
Anybody can help me? Thanks lot.
Re: [Streaming] How to pass arguments to a map/reduce script
Posted by Rong-en Fan <gr...@gmail.com>.
On Fri, Aug 22, 2008 at 12:51 AM, Steve Gao <st...@yahoo.com> wrote:
> That's interesting. Suppose your mapper script is a Perl script, how do you assign "my.mapper.arg1"'s value to a variable $x?
> $x = $my.mapper.arg1
> I just tried the way and my perl script does not recognize $my.mapper.arg1.
$ENV{my_mapper_arg1}
> --- On Thu, 8/21/08, Rong-en Fan <gr...@gmail.com> wrote:
> From: Rong-en Fan <gr...@gmail.com>
> Subject: Re: [Streaming] How to pass arguments to a map/reduce script
> To: core-user@hadoop.apache.org
> Cc: core-dev@hadoop.apache.org
> Date: Thursday, August 21, 2008, 11:09 AM
>
> On Thu, Aug 21, 2008 at 3:14 PM, Gopal Gandhi
> <go...@yahoo.com> wrote:
>> I am using Hadoop streaming and I need to pass arguments to my map/reduce
> script. Because a map/reduce script is triggered by hadoop, like
>> hadoop .... -file MAPPER -mapper "$MAPPER" -file REDUCER
> -reducer "$REDUCER" ...
>> How can I pass arguments to MAPPER?
>>
>> I tried -cmdenv name=val , but it does not work.
>> Anybody can help me? Thanks lot.
>
> I use -jobconf, for example
>
> hadoop ... -jobconf my.mapper.arg1="foobar"
>
> and in the map script, I get this by reading the environment variable
>
> my_mapper_arg1
>
> Hope this helps,
> Rong-En Fan
>>
>>
>>
>>
>
>
>
>
Re: [Streaming] How to pass arguments to a map/reduce script
Posted by Steve Gao <st...@yahoo.com>.
That's interesting. Suppose your mapper script is a Perl script, how do you assign "my.mapper.arg1"'s value to a variable $x?
$x = $my.mapper.arg1
I just tried the way and my perl script does not recognize $my.mapper.arg1.
--- On Thu, 8/21/08, Rong-en Fan <gr...@gmail.com> wrote:
From: Rong-en Fan <gr...@gmail.com>
Subject: Re: [Streaming] How to pass arguments to a map/reduce script
To: core-user@hadoop.apache.org
Cc: core-dev@hadoop.apache.org
Date: Thursday, August 21, 2008, 11:09 AM
On Thu, Aug 21, 2008 at 3:14 PM, Gopal Gandhi
<go...@yahoo.com> wrote:
> I am using Hadoop streaming and I need to pass arguments to my map/reduce
script. Because a map/reduce script is triggered by hadoop, like
> hadoop .... -file MAPPER -mapper "$MAPPER" -file REDUCER
-reducer "$REDUCER" ...
> How can I pass arguments to MAPPER?
>
> I tried -cmdenv name=val , but it does not work.
> Anybody can help me? Thanks lot.
I use -jobconf, for example
hadoop ... -jobconf my.mapper.arg1="foobar"
and in the map script, I get this by reading the environment variable
my_mapper_arg1
Hope this helps,
Rong-En Fan
>
>
>
>
Re: [Streaming] How to pass arguments to a map/reduce script
Posted by Rong-en Fan <gr...@gmail.com>.
On Thu, Aug 21, 2008 at 3:14 PM, Gopal Gandhi
<go...@yahoo.com> wrote:
> I am using Hadoop streaming and I need to pass arguments to my map/reduce script. Because a map/reduce script is triggered by hadoop, like
> hadoop .... -file MAPPER -mapper "$MAPPER" -file REDUCER -reducer "$REDUCER" ...
> How can I pass arguments to MAPPER?
>
> I tried -cmdenv name=val , but it does not work.
> Anybody can help me? Thanks lot.
I use -jobconf, for example
hadoop ... -jobconf my.mapper.arg1="foobar"
and in the map script, I get this by reading the environment variable
my_mapper_arg1
Hope this helps,
Rong-En Fan
>
>
>
>
Re: [Streaming] How to pass arguments to a map/reduce script
Posted by Karl Anderson <kr...@monkey.org>.
On 21-Aug-08, at 12:14 AM, Gopal Gandhi wrote:
> I am using Hadoop streaming and I need to pass arguments to my map/
> reduce script. Because a map/reduce script is triggered by hadoop,
> like
> hadoop .... -file MAPPER -mapper "$MAPPER" -file REDUCER -reducer
> "$REDUCER" ...
> How can I pass arguments to MAPPER?
>
> I tried -cmdenv name=val , but it does not work.
> Anybody can help me? Thanks lot.
-cmdenv works for me for Hadoop streaming 0.17.0 and 0.18.0. When my
job invocation ends with "-cmdenv FOO=bar", my Python mapper and
reducer scripts can get the value from the FOO environment variable.
Re: [Streaming] How to pass arguments to a map/reduce script
Posted by Rong-en Fan <gr...@gmail.com>.
On Thu, Aug 21, 2008 at 3:14 PM, Gopal Gandhi
<go...@yahoo.com> wrote:
> I am using Hadoop streaming and I need to pass arguments to my map/reduce script. Because a map/reduce script is triggered by hadoop, like
> hadoop .... -file MAPPER -mapper "$MAPPER" -file REDUCER -reducer "$REDUCER" ...
> How can I pass arguments to MAPPER?
>
> I tried -cmdenv name=val , but it does not work.
> Anybody can help me? Thanks lot.
I use -jobconf, for example
hadoop ... -jobconf my.mapper.arg1="foobar"
and in the map script, I get this by reading the environment variable
my_mapper_arg1
Hope this helps,
Rong-En Fan
>
>
>
>
Re: [Streaming] How to pass arguments to a map/reduce script
Posted by Yuri Pradkin <yu...@isi.edu>.
Don't forget the double quotes.
On Thursday 21 August 2008 11:06:10 Steve Gao wrote:
> Unfortunately this does not work. Hadoop complains:
> 08/08/21 18:04:46 ERROR streaming.StreamJob: Unexpected arg1 while
> processing
> -input|-output|-mapper|-combiner|-reducer|-file|-dfs|-jt|-additionalconfspe
>c|-inputformat|-outputformat|-partitioner|-numReduceTasks|-inputreader|-mapd
>ebug|-reducedebug|||-cacheFile|-cacheArchive|-verbose|-info|-debug|-inputtag
>ged|-help
> I think you can simply do:
> .... -file MAPPER -mapper "$MAPPER arg1 arg2" -file
> REDUCER -reducer "$REDUCER" ...
Re: [Streaming] How to pass arguments to a map/reduce script
Posted by Steve Gao <st...@yahoo.com>.
Unfortunately this does not work. Hadoop complains:
08/08/21 18:04:46 ERROR streaming.StreamJob: Unexpected arg1 while processing -input|-output|-mapper|-combiner|-reducer|-file|-dfs|-jt|-additionalconfspec|-inputformat|-outputformat|-partitioner|-numReduceTasks|-inputreader|-mapdebug|-reducedebug|||-cacheFile|-cacheArchive|-verbose|-info|-debug|-inputtagged|-help
--- On Thu, 8/21/08, Yuri Pradkin <yu...@isi.edu> wrote:
From: Yuri Pradkin <yu...@isi.edu>
Subject: Re: [Streaming] How to pass arguments to a map/reduce script
To: core-user@hadoop.apache.org
Cc: "Gopal Gandhi" <go...@yahoo.com>
Date: Thursday, August 21, 2008, 1:43 PM
On Thursday 21 August 2008 00:14:56 Gopal Gandhi wrote:
> I am using Hadoop streaming and I need to pass arguments to my map/reduce
> script. Because a map/reduce script is triggered by hadoop, like hadoop
> .... -file MAPPER -mapper "$MAPPER" -file REDUCER -reducer
"$REDUCER" ...
> How can I pass arguments to MAPPER?
>
> I tried -cmdenv name=val , but it does not work.
> Anybody can help me? Thanks lot.
I think you can simply do:
.... -file MAPPER -mapper "$MAPPER arg1 arg2" -file
REDUCER -reducer "$REDUCER" ...
Re: [Streaming] How to pass arguments to a map/reduce script
Posted by Yuri Pradkin <yu...@isi.edu>.
On Thursday 21 August 2008 00:14:56 Gopal Gandhi wrote:
> I am using Hadoop streaming and I need to pass arguments to my map/reduce
> script. Because a map/reduce script is triggered by hadoop, like hadoop
> .... -file MAPPER -mapper "$MAPPER" -file REDUCER -reducer "$REDUCER" ...
> How can I pass arguments to MAPPER?
>
> I tried -cmdenv name=val , but it does not work.
> Anybody can help me? Thanks lot.
I think you can simply do:
.... -file MAPPER -mapper "$MAPPER arg1 arg2" -file
REDUCER -reducer "$REDUCER" ...
Re: [Streaming] How to pass arguments to a map/reduce script
Posted by arkady borkovsky <ar...@yahoo-inc.com>.
The values of
-mapper
and
-reducer
are a command lines -- just pu the parameters there
-ab
On Aug 21, 2008, at 12:14 AM, Gopal Gandhi wrote:
> I am using Hadoop streaming and I need to pass arguments to my map/
> reduce script. Because a map/reduce script is triggered by hadoop,
> like
> hadoop .... -file MAPPER -mapper "$MAPPER" -file REDUCER -reducer
> "$REDUCER" ...
> How can I pass arguments to MAPPER?
>
> I tried -cmdenv name=val , but it does not work.
> Anybody can help me? Thanks lot.
>
>
>