You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Istabrak Abdul-Fatah <if...@gmail.com> on 2015/10/06 15:22:39 UTC
Passing Args to Mappers and Reducers
Greetings to all,
Is it possible to pass args to Mapper and Reducers via the command line
args.
I need to pass some args upon the MapRed job invocation so that I can pass
these args via the Context object to the Mapper and Reducer code.
I am currently running Hadoop2.7 and tried to pass some args using the -D
VM args but it did not succeed.
Here are some failed code snippets:
~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
/user/yarn/input/samplefile.csv output
~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
/user/yarn/input/samplefile.csv output
~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
/user/yarn/input/samplefile.csv output -Dista="666666"
Please advise.
Thx and BR
Ista
Re: Passing Args to Mappers and Reducers
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Varun,
Thx for the info.
I got it fixed and successfully got the passed-in property val from the
conf object rather than from the
System.getProperty().
Regards,
Ista
On Tue, Oct 6, 2015 at 12:06 PM, Varun Saxena <vs...@gmail.com>
wrote:
> Hi Istabrak,
>
> Sorry had misunderstood your query a little.
> The code written is a little bit wrong. You are not really using the Tool
> class.
> You can change the code as under :
>
> public static void main(String[] argv) throws Exception {
> int ret = ToolRunner.run(null, new AvgSix(), argv));
> System.exit(ret);
> }
>
> You can then put your main code in a method named run (which overrides
> Tool#run)
> yarn jar will forward whatever you pass from command line to
> Tool#run(String[] args), in your case AvgSix#run
>
> You can either choose to pass all the parameters from the command line and
> parse them in run method yourself or you can choose to pass it with -D
> option.
> When you use -D option the property mapping will be available in conf
> object. As you can see you are extending Configured class as well.
> So if you run a command like
>
> yarn jar /opt/yarn/my_examples/AvgSix.jar *-Dista=667788*
> /user/yarn/input/samplePMfile.v31.csv output
>
> Please note all the -D params should be before other arguments.
>
> You can then get the value of property named ista in AvgSix#run as under :
> Configuration conf = getConf();
> String propertyVal = conf.get("ista");
>
> This is the basic idea. You can change the code as per your requirement.
>
> Regards,
> Varun.
>
>
> On Tue, Oct 6, 2015 at 8:49 PM, Istabrak Abdul-Fatah <if...@gmail.com>
> wrote:
>
>> Hi Naga,
>> Thx for the follow up.
>> Still not working.
>> Here are the invocation errors and the source code.
>>
>> [image: Inline image 1]
>>
>>
>> public class AvgSix extends Configured implements Tool{
>>
>> public static void main(String[] args) throws Exception {
>> Configuration conf = new Configuration();
>> conf.set("test", System.getProperty("ista")); // <=== This is the line
>> where it is failing
>> int res = ToolRunner.run(conf, new AvgSix(), args);
>> System.exit(res);
>> }
>>
>> [yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>> -Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
>> thread "main" java.lang.IllegalArgumentException: The value of property
>> test must not be null at
>> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
>> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
>> AvgSix.main(AvgSix.java:32) at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497) at
>> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
>> org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881
>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
>> /user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
>> java.lang.IllegalArgumentException: The value of property test must not be
>> null at
>> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
>> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
>> AvgSix.main(AvgSix.java:32) at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497) at
>> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
>> org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>
>>
>> On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
>> garlanaganarasimha@huawei.com> wrote:
>>
>>> Hi Ista,
>>>
>>> In general we need to give after specifying the class name like
>>> *./yarn jar
>>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>>
>>> I think in your case it should be "*yarn jar
>>> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
>>> /user/yarn/input/samplefile.csv output*"
>>>
>>> + Naga
>>> ------------------------------
>>> *From:* Naganarasimha G R (Naga)
>>> *Sent:* Tuesday, October 06, 2015 20:16
>>> *To:* user@hadoop.apache.org
>>> *Subject:* RE: Passing Args to Mappers and Reducers
>>>
>>> Hi Ista,
>>>
>>> In general we need to give after specifying the class name like
>>> *./yarn jar
>>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>>
>>> i think in your case it should be
>>> ------------------------------
>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>> *Sent:* Tuesday, October 06, 2015 19:46
>>> *To:* user@hadoop.apache.org
>>> *Subject:* Re: Passing Args to Mappers and Reducers
>>>
>>> Hi Shahab,
>>> Thx for the quick reply.
>>> Yes, I am using the tool interface and can run the job successfully.
>>> I need to pass some extra args dynamically as part of the invocation.
>>>
>>> I have tried adding a space after the -D option and it did not work
>>> either (see below).
>>>
>>> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/sample.csv output
>>> Error: Could not find or load main class ista=666666
>>>
>>> It does not see to recognize the -D option.
>>>
>>> BR
>>>
>>> Ista
>>>
>>> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
>>> wrote:
>>>
>>>> Are you properly implementing the Tool interface?
>>>>
>>>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>>>
>>>> Also, there needs to be space between -D and the param name.
>>>>
>>>> Regards,
>>>> Shahab
>>>>
>>>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>>>> wrote:
>>>>
>>>>> Greetings to all,
>>>>> Is it possible to pass args to Mapper and Reducers via the command
>>>>> line args.
>>>>> I need to pass some args upon the MapRed job invocation so that I can
>>>>> pass these args via the Context object to the Mapper and Reducer code.
>>>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>>>> -D VM args but it did not succeed.
>>>>> Here are some failed code snippets:
>>>>>
>>>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>>>> /user/yarn/input/samplefile.csv output
>>>>>
>>>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>>>> /user/yarn/input/samplefile.csv output
>>>>>
>>>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>>>
>>>>> Please advise.
>>>>>
>>>>>
>>>>> Thx and BR
>>>>>
>>>>> Ista
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Varun,
Thx for the info.
I got it fixed and successfully got the passed-in property val from the
conf object rather than from the
System.getProperty().
Regards,
Ista
On Tue, Oct 6, 2015 at 12:06 PM, Varun Saxena <vs...@gmail.com>
wrote:
> Hi Istabrak,
>
> Sorry had misunderstood your query a little.
> The code written is a little bit wrong. You are not really using the Tool
> class.
> You can change the code as under :
>
> public static void main(String[] argv) throws Exception {
> int ret = ToolRunner.run(null, new AvgSix(), argv));
> System.exit(ret);
> }
>
> You can then put your main code in a method named run (which overrides
> Tool#run)
> yarn jar will forward whatever you pass from command line to
> Tool#run(String[] args), in your case AvgSix#run
>
> You can either choose to pass all the parameters from the command line and
> parse them in run method yourself or you can choose to pass it with -D
> option.
> When you use -D option the property mapping will be available in conf
> object. As you can see you are extending Configured class as well.
> So if you run a command like
>
> yarn jar /opt/yarn/my_examples/AvgSix.jar *-Dista=667788*
> /user/yarn/input/samplePMfile.v31.csv output
>
> Please note all the -D params should be before other arguments.
>
> You can then get the value of property named ista in AvgSix#run as under :
> Configuration conf = getConf();
> String propertyVal = conf.get("ista");
>
> This is the basic idea. You can change the code as per your requirement.
>
> Regards,
> Varun.
>
>
> On Tue, Oct 6, 2015 at 8:49 PM, Istabrak Abdul-Fatah <if...@gmail.com>
> wrote:
>
>> Hi Naga,
>> Thx for the follow up.
>> Still not working.
>> Here are the invocation errors and the source code.
>>
>> [image: Inline image 1]
>>
>>
>> public class AvgSix extends Configured implements Tool{
>>
>> public static void main(String[] args) throws Exception {
>> Configuration conf = new Configuration();
>> conf.set("test", System.getProperty("ista")); // <=== This is the line
>> where it is failing
>> int res = ToolRunner.run(conf, new AvgSix(), args);
>> System.exit(res);
>> }
>>
>> [yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>> -Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
>> thread "main" java.lang.IllegalArgumentException: The value of property
>> test must not be null at
>> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
>> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
>> AvgSix.main(AvgSix.java:32) at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497) at
>> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
>> org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881
>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
>> /user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
>> java.lang.IllegalArgumentException: The value of property test must not be
>> null at
>> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
>> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
>> AvgSix.main(AvgSix.java:32) at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497) at
>> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
>> org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>
>>
>> On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
>> garlanaganarasimha@huawei.com> wrote:
>>
>>> Hi Ista,
>>>
>>> In general we need to give after specifying the class name like
>>> *./yarn jar
>>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>>
>>> I think in your case it should be "*yarn jar
>>> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
>>> /user/yarn/input/samplefile.csv output*"
>>>
>>> + Naga
>>> ------------------------------
>>> *From:* Naganarasimha G R (Naga)
>>> *Sent:* Tuesday, October 06, 2015 20:16
>>> *To:* user@hadoop.apache.org
>>> *Subject:* RE: Passing Args to Mappers and Reducers
>>>
>>> Hi Ista,
>>>
>>> In general we need to give after specifying the class name like
>>> *./yarn jar
>>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>>
>>> i think in your case it should be
>>> ------------------------------
>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>> *Sent:* Tuesday, October 06, 2015 19:46
>>> *To:* user@hadoop.apache.org
>>> *Subject:* Re: Passing Args to Mappers and Reducers
>>>
>>> Hi Shahab,
>>> Thx for the quick reply.
>>> Yes, I am using the tool interface and can run the job successfully.
>>> I need to pass some extra args dynamically as part of the invocation.
>>>
>>> I have tried adding a space after the -D option and it did not work
>>> either (see below).
>>>
>>> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/sample.csv output
>>> Error: Could not find or load main class ista=666666
>>>
>>> It does not see to recognize the -D option.
>>>
>>> BR
>>>
>>> Ista
>>>
>>> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
>>> wrote:
>>>
>>>> Are you properly implementing the Tool interface?
>>>>
>>>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>>>
>>>> Also, there needs to be space between -D and the param name.
>>>>
>>>> Regards,
>>>> Shahab
>>>>
>>>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>>>> wrote:
>>>>
>>>>> Greetings to all,
>>>>> Is it possible to pass args to Mapper and Reducers via the command
>>>>> line args.
>>>>> I need to pass some args upon the MapRed job invocation so that I can
>>>>> pass these args via the Context object to the Mapper and Reducer code.
>>>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>>>> -D VM args but it did not succeed.
>>>>> Here are some failed code snippets:
>>>>>
>>>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>>>> /user/yarn/input/samplefile.csv output
>>>>>
>>>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>>>> /user/yarn/input/samplefile.csv output
>>>>>
>>>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>>>
>>>>> Please advise.
>>>>>
>>>>>
>>>>> Thx and BR
>>>>>
>>>>> Ista
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Varun,
Thx for the info.
I got it fixed and successfully got the passed-in property val from the
conf object rather than from the
System.getProperty().
Regards,
Ista
On Tue, Oct 6, 2015 at 12:06 PM, Varun Saxena <vs...@gmail.com>
wrote:
> Hi Istabrak,
>
> Sorry had misunderstood your query a little.
> The code written is a little bit wrong. You are not really using the Tool
> class.
> You can change the code as under :
>
> public static void main(String[] argv) throws Exception {
> int ret = ToolRunner.run(null, new AvgSix(), argv));
> System.exit(ret);
> }
>
> You can then put your main code in a method named run (which overrides
> Tool#run)
> yarn jar will forward whatever you pass from command line to
> Tool#run(String[] args), in your case AvgSix#run
>
> You can either choose to pass all the parameters from the command line and
> parse them in run method yourself or you can choose to pass it with -D
> option.
> When you use -D option the property mapping will be available in conf
> object. As you can see you are extending Configured class as well.
> So if you run a command like
>
> yarn jar /opt/yarn/my_examples/AvgSix.jar *-Dista=667788*
> /user/yarn/input/samplePMfile.v31.csv output
>
> Please note all the -D params should be before other arguments.
>
> You can then get the value of property named ista in AvgSix#run as under :
> Configuration conf = getConf();
> String propertyVal = conf.get("ista");
>
> This is the basic idea. You can change the code as per your requirement.
>
> Regards,
> Varun.
>
>
> On Tue, Oct 6, 2015 at 8:49 PM, Istabrak Abdul-Fatah <if...@gmail.com>
> wrote:
>
>> Hi Naga,
>> Thx for the follow up.
>> Still not working.
>> Here are the invocation errors and the source code.
>>
>> [image: Inline image 1]
>>
>>
>> public class AvgSix extends Configured implements Tool{
>>
>> public static void main(String[] args) throws Exception {
>> Configuration conf = new Configuration();
>> conf.set("test", System.getProperty("ista")); // <=== This is the line
>> where it is failing
>> int res = ToolRunner.run(conf, new AvgSix(), args);
>> System.exit(res);
>> }
>>
>> [yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>> -Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
>> thread "main" java.lang.IllegalArgumentException: The value of property
>> test must not be null at
>> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
>> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
>> AvgSix.main(AvgSix.java:32) at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497) at
>> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
>> org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881
>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
>> /user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
>> java.lang.IllegalArgumentException: The value of property test must not be
>> null at
>> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
>> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
>> AvgSix.main(AvgSix.java:32) at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497) at
>> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
>> org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>
>>
>> On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
>> garlanaganarasimha@huawei.com> wrote:
>>
>>> Hi Ista,
>>>
>>> In general we need to give after specifying the class name like
>>> *./yarn jar
>>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>>
>>> I think in your case it should be "*yarn jar
>>> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
>>> /user/yarn/input/samplefile.csv output*"
>>>
>>> + Naga
>>> ------------------------------
>>> *From:* Naganarasimha G R (Naga)
>>> *Sent:* Tuesday, October 06, 2015 20:16
>>> *To:* user@hadoop.apache.org
>>> *Subject:* RE: Passing Args to Mappers and Reducers
>>>
>>> Hi Ista,
>>>
>>> In general we need to give after specifying the class name like
>>> *./yarn jar
>>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>>
>>> i think in your case it should be
>>> ------------------------------
>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>> *Sent:* Tuesday, October 06, 2015 19:46
>>> *To:* user@hadoop.apache.org
>>> *Subject:* Re: Passing Args to Mappers and Reducers
>>>
>>> Hi Shahab,
>>> Thx for the quick reply.
>>> Yes, I am using the tool interface and can run the job successfully.
>>> I need to pass some extra args dynamically as part of the invocation.
>>>
>>> I have tried adding a space after the -D option and it did not work
>>> either (see below).
>>>
>>> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/sample.csv output
>>> Error: Could not find or load main class ista=666666
>>>
>>> It does not see to recognize the -D option.
>>>
>>> BR
>>>
>>> Ista
>>>
>>> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
>>> wrote:
>>>
>>>> Are you properly implementing the Tool interface?
>>>>
>>>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>>>
>>>> Also, there needs to be space between -D and the param name.
>>>>
>>>> Regards,
>>>> Shahab
>>>>
>>>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>>>> wrote:
>>>>
>>>>> Greetings to all,
>>>>> Is it possible to pass args to Mapper and Reducers via the command
>>>>> line args.
>>>>> I need to pass some args upon the MapRed job invocation so that I can
>>>>> pass these args via the Context object to the Mapper and Reducer code.
>>>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>>>> -D VM args but it did not succeed.
>>>>> Here are some failed code snippets:
>>>>>
>>>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>>>> /user/yarn/input/samplefile.csv output
>>>>>
>>>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>>>> /user/yarn/input/samplefile.csv output
>>>>>
>>>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>>>
>>>>> Please advise.
>>>>>
>>>>>
>>>>> Thx and BR
>>>>>
>>>>> Ista
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Varun,
Thx for the info.
I got it fixed and successfully got the passed-in property val from the
conf object rather than from the
System.getProperty().
Regards,
Ista
On Tue, Oct 6, 2015 at 12:06 PM, Varun Saxena <vs...@gmail.com>
wrote:
> Hi Istabrak,
>
> Sorry had misunderstood your query a little.
> The code written is a little bit wrong. You are not really using the Tool
> class.
> You can change the code as under :
>
> public static void main(String[] argv) throws Exception {
> int ret = ToolRunner.run(null, new AvgSix(), argv));
> System.exit(ret);
> }
>
> You can then put your main code in a method named run (which overrides
> Tool#run)
> yarn jar will forward whatever you pass from command line to
> Tool#run(String[] args), in your case AvgSix#run
>
> You can either choose to pass all the parameters from the command line and
> parse them in run method yourself or you can choose to pass it with -D
> option.
> When you use -D option the property mapping will be available in conf
> object. As you can see you are extending Configured class as well.
> So if you run a command like
>
> yarn jar /opt/yarn/my_examples/AvgSix.jar *-Dista=667788*
> /user/yarn/input/samplePMfile.v31.csv output
>
> Please note all the -D params should be before other arguments.
>
> You can then get the value of property named ista in AvgSix#run as under :
> Configuration conf = getConf();
> String propertyVal = conf.get("ista");
>
> This is the basic idea. You can change the code as per your requirement.
>
> Regards,
> Varun.
>
>
> On Tue, Oct 6, 2015 at 8:49 PM, Istabrak Abdul-Fatah <if...@gmail.com>
> wrote:
>
>> Hi Naga,
>> Thx for the follow up.
>> Still not working.
>> Here are the invocation errors and the source code.
>>
>> [image: Inline image 1]
>>
>>
>> public class AvgSix extends Configured implements Tool{
>>
>> public static void main(String[] args) throws Exception {
>> Configuration conf = new Configuration();
>> conf.set("test", System.getProperty("ista")); // <=== This is the line
>> where it is failing
>> int res = ToolRunner.run(conf, new AvgSix(), args);
>> System.exit(res);
>> }
>>
>> [yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>> -Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
>> thread "main" java.lang.IllegalArgumentException: The value of property
>> test must not be null at
>> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
>> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
>> AvgSix.main(AvgSix.java:32) at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497) at
>> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
>> org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881
>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
>> /user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
>> java.lang.IllegalArgumentException: The value of property test must not be
>> null at
>> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
>> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
>> AvgSix.main(AvgSix.java:32) at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497) at
>> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
>> org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>
>>
>> On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
>> garlanaganarasimha@huawei.com> wrote:
>>
>>> Hi Ista,
>>>
>>> In general we need to give after specifying the class name like
>>> *./yarn jar
>>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>>
>>> I think in your case it should be "*yarn jar
>>> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
>>> /user/yarn/input/samplefile.csv output*"
>>>
>>> + Naga
>>> ------------------------------
>>> *From:* Naganarasimha G R (Naga)
>>> *Sent:* Tuesday, October 06, 2015 20:16
>>> *To:* user@hadoop.apache.org
>>> *Subject:* RE: Passing Args to Mappers and Reducers
>>>
>>> Hi Ista,
>>>
>>> In general we need to give after specifying the class name like
>>> *./yarn jar
>>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>>
>>> i think in your case it should be
>>> ------------------------------
>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>> *Sent:* Tuesday, October 06, 2015 19:46
>>> *To:* user@hadoop.apache.org
>>> *Subject:* Re: Passing Args to Mappers and Reducers
>>>
>>> Hi Shahab,
>>> Thx for the quick reply.
>>> Yes, I am using the tool interface and can run the job successfully.
>>> I need to pass some extra args dynamically as part of the invocation.
>>>
>>> I have tried adding a space after the -D option and it did not work
>>> either (see below).
>>>
>>> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/sample.csv output
>>> Error: Could not find or load main class ista=666666
>>>
>>> It does not see to recognize the -D option.
>>>
>>> BR
>>>
>>> Ista
>>>
>>> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
>>> wrote:
>>>
>>>> Are you properly implementing the Tool interface?
>>>>
>>>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>>>
>>>> Also, there needs to be space between -D and the param name.
>>>>
>>>> Regards,
>>>> Shahab
>>>>
>>>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>>>> wrote:
>>>>
>>>>> Greetings to all,
>>>>> Is it possible to pass args to Mapper and Reducers via the command
>>>>> line args.
>>>>> I need to pass some args upon the MapRed job invocation so that I can
>>>>> pass these args via the Context object to the Mapper and Reducer code.
>>>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>>>> -D VM args but it did not succeed.
>>>>> Here are some failed code snippets:
>>>>>
>>>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>>>> /user/yarn/input/samplefile.csv output
>>>>>
>>>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>>>> /user/yarn/input/samplefile.csv output
>>>>>
>>>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>>>
>>>>> Please advise.
>>>>>
>>>>>
>>>>> Thx and BR
>>>>>
>>>>> Ista
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Varun Saxena <vs...@gmail.com>.
Hi Istabrak,
Sorry had misunderstood your query a little.
The code written is a little bit wrong. You are not really using the Tool
class.
You can change the code as under :
public static void main(String[] argv) throws Exception {
int ret = ToolRunner.run(null, new AvgSix(), argv));
System.exit(ret);
}
You can then put your main code in a method named run (which overrides
Tool#run)
yarn jar will forward whatever you pass from command line to
Tool#run(String[] args), in your case AvgSix#run
You can either choose to pass all the parameters from the command line and
parse them in run method yourself or you can choose to pass it with -D
option.
When you use -D option the property mapping will be available in conf
object. As you can see you are extending Configured class as well.
So if you run a command like
yarn jar /opt/yarn/my_examples/AvgSix.jar *-Dista=667788*
/user/yarn/input/samplePMfile.v31.csv output
Please note all the -D params should be before other arguments.
You can then get the value of property named ista in AvgSix#run as under :
Configuration conf = getConf();
String propertyVal = conf.get("ista");
This is the basic idea. You can change the code as per your requirement.
Regards,
Varun.
On Tue, Oct 6, 2015 at 8:49 PM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Hi Naga,
> Thx for the follow up.
> Still not working.
> Here are the invocation errors and the source code.
>
> [image: Inline image 1]
>
>
> public class AvgSix extends Configured implements Tool{
>
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("test", System.getProperty("ista")); // <=== This is the line
> where it is failing
> int res = ToolRunner.run(conf, new AvgSix(), args);
> System.exit(res);
> }
>
> [yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
> -Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
> thread "main" java.lang.IllegalArgumentException: The value of property
> test must not be null at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
> AvgSix.main(AvgSix.java:32) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881
> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
> /user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
> java.lang.IllegalArgumentException: The value of property test must not be
> null at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
> AvgSix.main(AvgSix.java:32) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
>
> On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
> garlanaganarasimha@huawei.com> wrote:
>
>> Hi Ista,
>>
>> In general we need to give after specifying the class name like
>> *./yarn jar
>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>
>> I think in your case it should be "*yarn jar
>> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
>> /user/yarn/input/samplefile.csv output*"
>>
>> + Naga
>> ------------------------------
>> *From:* Naganarasimha G R (Naga)
>> *Sent:* Tuesday, October 06, 2015 20:16
>> *To:* user@hadoop.apache.org
>> *Subject:* RE: Passing Args to Mappers and Reducers
>>
>> Hi Ista,
>>
>> In general we need to give after specifying the class name like
>> *./yarn jar
>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>
>> i think in your case it should be
>> ------------------------------
>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>> *Sent:* Tuesday, October 06, 2015 19:46
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Passing Args to Mappers and Reducers
>>
>> Hi Shahab,
>> Thx for the quick reply.
>> Yes, I am using the tool interface and can run the job successfully.
>> I need to pass some extra args dynamically as part of the invocation.
>>
>> I have tried adding a space after the -D option and it did not work
>> either (see below).
>>
>> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/sample.csv output
>> Error: Could not find or load main class ista=666666
>>
>> It does not see to recognize the -D option.
>>
>> BR
>>
>> Ista
>>
>> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
>> wrote:
>>
>>> Are you properly implementing the Tool interface?
>>>
>>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>>
>>> Also, there needs to be space between -D and the param name.
>>>
>>> Regards,
>>> Shahab
>>>
>>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>>> wrote:
>>>
>>>> Greetings to all,
>>>> Is it possible to pass args to Mapper and Reducers via the command line
>>>> args.
>>>> I need to pass some args upon the MapRed job invocation so that I can
>>>> pass these args via the Context object to the Mapper and Reducer code.
>>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>>> -D VM args but it did not succeed.
>>>> Here are some failed code snippets:
>>>>
>>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>>> /user/yarn/input/samplefile.csv output
>>>>
>>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>>> /user/yarn/input/samplefile.csv output
>>>>
>>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>>
>>>> Please advise.
>>>>
>>>>
>>>> Thx and BR
>>>>
>>>> Ista
>>>>
>>>
>>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Varun Saxena <vs...@gmail.com>.
Hi Istabrak,
Sorry had misunderstood your query a little.
The code written is a little bit wrong. You are not really using the Tool
class.
You can change the code as under :
public static void main(String[] argv) throws Exception {
int ret = ToolRunner.run(null, new AvgSix(), argv));
System.exit(ret);
}
You can then put your main code in a method named run (which overrides
Tool#run)
yarn jar will forward whatever you pass from command line to
Tool#run(String[] args), in your case AvgSix#run
You can either choose to pass all the parameters from the command line and
parse them in run method yourself or you can choose to pass it with -D
option.
When you use -D option the property mapping will be available in conf
object. As you can see you are extending Configured class as well.
So if you run a command like
yarn jar /opt/yarn/my_examples/AvgSix.jar *-Dista=667788*
/user/yarn/input/samplePMfile.v31.csv output
Please note all the -D params should be before other arguments.
You can then get the value of property named ista in AvgSix#run as under :
Configuration conf = getConf();
String propertyVal = conf.get("ista");
This is the basic idea. You can change the code as per your requirement.
Regards,
Varun.
On Tue, Oct 6, 2015 at 8:49 PM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Hi Naga,
> Thx for the follow up.
> Still not working.
> Here are the invocation errors and the source code.
>
> [image: Inline image 1]
>
>
> public class AvgSix extends Configured implements Tool{
>
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("test", System.getProperty("ista")); // <=== This is the line
> where it is failing
> int res = ToolRunner.run(conf, new AvgSix(), args);
> System.exit(res);
> }
>
> [yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
> -Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
> thread "main" java.lang.IllegalArgumentException: The value of property
> test must not be null at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
> AvgSix.main(AvgSix.java:32) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881
> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
> /user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
> java.lang.IllegalArgumentException: The value of property test must not be
> null at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
> AvgSix.main(AvgSix.java:32) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
>
> On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
> garlanaganarasimha@huawei.com> wrote:
>
>> Hi Ista,
>>
>> In general we need to give after specifying the class name like
>> *./yarn jar
>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>
>> I think in your case it should be "*yarn jar
>> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
>> /user/yarn/input/samplefile.csv output*"
>>
>> + Naga
>> ------------------------------
>> *From:* Naganarasimha G R (Naga)
>> *Sent:* Tuesday, October 06, 2015 20:16
>> *To:* user@hadoop.apache.org
>> *Subject:* RE: Passing Args to Mappers and Reducers
>>
>> Hi Ista,
>>
>> In general we need to give after specifying the class name like
>> *./yarn jar
>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>
>> i think in your case it should be
>> ------------------------------
>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>> *Sent:* Tuesday, October 06, 2015 19:46
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Passing Args to Mappers and Reducers
>>
>> Hi Shahab,
>> Thx for the quick reply.
>> Yes, I am using the tool interface and can run the job successfully.
>> I need to pass some extra args dynamically as part of the invocation.
>>
>> I have tried adding a space after the -D option and it did not work
>> either (see below).
>>
>> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/sample.csv output
>> Error: Could not find or load main class ista=666666
>>
>> It does not see to recognize the -D option.
>>
>> BR
>>
>> Ista
>>
>> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
>> wrote:
>>
>>> Are you properly implementing the Tool interface?
>>>
>>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>>
>>> Also, there needs to be space between -D and the param name.
>>>
>>> Regards,
>>> Shahab
>>>
>>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>>> wrote:
>>>
>>>> Greetings to all,
>>>> Is it possible to pass args to Mapper and Reducers via the command line
>>>> args.
>>>> I need to pass some args upon the MapRed job invocation so that I can
>>>> pass these args via the Context object to the Mapper and Reducer code.
>>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>>> -D VM args but it did not succeed.
>>>> Here are some failed code snippets:
>>>>
>>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>>> /user/yarn/input/samplefile.csv output
>>>>
>>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>>> /user/yarn/input/samplefile.csv output
>>>>
>>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>>
>>>> Please advise.
>>>>
>>>>
>>>> Thx and BR
>>>>
>>>> Ista
>>>>
>>>
>>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Varun Saxena <vs...@gmail.com>.
Hi Istabrak,
Sorry had misunderstood your query a little.
The code written is a little bit wrong. You are not really using the Tool
class.
You can change the code as under :
public static void main(String[] argv) throws Exception {
int ret = ToolRunner.run(null, new AvgSix(), argv));
System.exit(ret);
}
You can then put your main code in a method named run (which overrides
Tool#run)
yarn jar will forward whatever you pass from command line to
Tool#run(String[] args), in your case AvgSix#run
You can either choose to pass all the parameters from the command line and
parse them in run method yourself or you can choose to pass it with -D
option.
When you use -D option the property mapping will be available in conf
object. As you can see you are extending Configured class as well.
So if you run a command like
yarn jar /opt/yarn/my_examples/AvgSix.jar *-Dista=667788*
/user/yarn/input/samplePMfile.v31.csv output
Please note all the -D params should be before other arguments.
You can then get the value of property named ista in AvgSix#run as under :
Configuration conf = getConf();
String propertyVal = conf.get("ista");
This is the basic idea. You can change the code as per your requirement.
Regards,
Varun.
On Tue, Oct 6, 2015 at 8:49 PM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Hi Naga,
> Thx for the follow up.
> Still not working.
> Here are the invocation errors and the source code.
>
> [image: Inline image 1]
>
>
> public class AvgSix extends Configured implements Tool{
>
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("test", System.getProperty("ista")); // <=== This is the line
> where it is failing
> int res = ToolRunner.run(conf, new AvgSix(), args);
> System.exit(res);
> }
>
> [yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
> -Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
> thread "main" java.lang.IllegalArgumentException: The value of property
> test must not be null at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
> AvgSix.main(AvgSix.java:32) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881
> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
> /user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
> java.lang.IllegalArgumentException: The value of property test must not be
> null at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
> AvgSix.main(AvgSix.java:32) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
>
> On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
> garlanaganarasimha@huawei.com> wrote:
>
>> Hi Ista,
>>
>> In general we need to give after specifying the class name like
>> *./yarn jar
>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>
>> I think in your case it should be "*yarn jar
>> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
>> /user/yarn/input/samplefile.csv output*"
>>
>> + Naga
>> ------------------------------
>> *From:* Naganarasimha G R (Naga)
>> *Sent:* Tuesday, October 06, 2015 20:16
>> *To:* user@hadoop.apache.org
>> *Subject:* RE: Passing Args to Mappers and Reducers
>>
>> Hi Ista,
>>
>> In general we need to give after specifying the class name like
>> *./yarn jar
>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>
>> i think in your case it should be
>> ------------------------------
>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>> *Sent:* Tuesday, October 06, 2015 19:46
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Passing Args to Mappers and Reducers
>>
>> Hi Shahab,
>> Thx for the quick reply.
>> Yes, I am using the tool interface and can run the job successfully.
>> I need to pass some extra args dynamically as part of the invocation.
>>
>> I have tried adding a space after the -D option and it did not work
>> either (see below).
>>
>> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/sample.csv output
>> Error: Could not find or load main class ista=666666
>>
>> It does not see to recognize the -D option.
>>
>> BR
>>
>> Ista
>>
>> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
>> wrote:
>>
>>> Are you properly implementing the Tool interface?
>>>
>>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>>
>>> Also, there needs to be space between -D and the param name.
>>>
>>> Regards,
>>> Shahab
>>>
>>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>>> wrote:
>>>
>>>> Greetings to all,
>>>> Is it possible to pass args to Mapper and Reducers via the command line
>>>> args.
>>>> I need to pass some args upon the MapRed job invocation so that I can
>>>> pass these args via the Context object to the Mapper and Reducer code.
>>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>>> -D VM args but it did not succeed.
>>>> Here are some failed code snippets:
>>>>
>>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>>> /user/yarn/input/samplefile.csv output
>>>>
>>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>>> /user/yarn/input/samplefile.csv output
>>>>
>>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>>
>>>> Please advise.
>>>>
>>>>
>>>> Thx and BR
>>>>
>>>> Ista
>>>>
>>>
>>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Varun Saxena <vs...@gmail.com>.
Hi Istabrak,
Sorry had misunderstood your query a little.
The code written is a little bit wrong. You are not really using the Tool
class.
You can change the code as under :
public static void main(String[] argv) throws Exception {
int ret = ToolRunner.run(null, new AvgSix(), argv));
System.exit(ret);
}
You can then put your main code in a method named run (which overrides
Tool#run)
yarn jar will forward whatever you pass from command line to
Tool#run(String[] args), in your case AvgSix#run
You can either choose to pass all the parameters from the command line and
parse them in run method yourself or you can choose to pass it with -D
option.
When you use -D option the property mapping will be available in conf
object. As you can see you are extending Configured class as well.
So if you run a command like
yarn jar /opt/yarn/my_examples/AvgSix.jar *-Dista=667788*
/user/yarn/input/samplePMfile.v31.csv output
Please note all the -D params should be before other arguments.
You can then get the value of property named ista in AvgSix#run as under :
Configuration conf = getConf();
String propertyVal = conf.get("ista");
This is the basic idea. You can change the code as per your requirement.
Regards,
Varun.
On Tue, Oct 6, 2015 at 8:49 PM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Hi Naga,
> Thx for the follow up.
> Still not working.
> Here are the invocation errors and the source code.
>
> [image: Inline image 1]
>
>
> public class AvgSix extends Configured implements Tool{
>
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("test", System.getProperty("ista")); // <=== This is the line
> where it is failing
> int res = ToolRunner.run(conf, new AvgSix(), args);
> System.exit(res);
> }
>
> [yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
> -Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
> thread "main" java.lang.IllegalArgumentException: The value of property
> test must not be null at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
> AvgSix.main(AvgSix.java:32) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881
> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
> /user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
> java.lang.IllegalArgumentException: The value of property test must not be
> null at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
> AvgSix.main(AvgSix.java:32) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
>
> On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
> garlanaganarasimha@huawei.com> wrote:
>
>> Hi Ista,
>>
>> In general we need to give after specifying the class name like
>> *./yarn jar
>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>
>> I think in your case it should be "*yarn jar
>> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
>> /user/yarn/input/samplefile.csv output*"
>>
>> + Naga
>> ------------------------------
>> *From:* Naganarasimha G R (Naga)
>> *Sent:* Tuesday, October 06, 2015 20:16
>> *To:* user@hadoop.apache.org
>> *Subject:* RE: Passing Args to Mappers and Reducers
>>
>> Hi Ista,
>>
>> In general we need to give after specifying the class name like
>> *./yarn jar
>> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
>> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>>
>> i think in your case it should be
>> ------------------------------
>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>> *Sent:* Tuesday, October 06, 2015 19:46
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Passing Args to Mappers and Reducers
>>
>> Hi Shahab,
>> Thx for the quick reply.
>> Yes, I am using the tool interface and can run the job successfully.
>> I need to pass some extra args dynamically as part of the invocation.
>>
>> I have tried adding a space after the -D option and it did not work
>> either (see below).
>>
>> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/sample.csv output
>> Error: Could not find or load main class ista=666666
>>
>> It does not see to recognize the -D option.
>>
>> BR
>>
>> Ista
>>
>> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
>> wrote:
>>
>>> Are you properly implementing the Tool interface?
>>>
>>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>>
>>> Also, there needs to be space between -D and the param name.
>>>
>>> Regards,
>>> Shahab
>>>
>>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>>> wrote:
>>>
>>>> Greetings to all,
>>>> Is it possible to pass args to Mapper and Reducers via the command line
>>>> args.
>>>> I need to pass some args upon the MapRed job invocation so that I can
>>>> pass these args via the Context object to the Mapper and Reducer code.
>>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>>> -D VM args but it did not succeed.
>>>> Here are some failed code snippets:
>>>>
>>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>>> /user/yarn/input/samplefile.csv output
>>>>
>>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>>> /user/yarn/input/samplefile.csv output
>>>>
>>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>>
>>>> Please advise.
>>>>
>>>>
>>>> Thx and BR
>>>>
>>>> Ista
>>>>
>>>
>>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Naga,
Thx for the follow up.
Still not working.
Here are the invocation errors and the source code.
[image: Inline image 1]
public class AvgSix extends Configured implements Tool{
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("test", System.getProperty("ista")); // <=== This is the line
where it is failing
int res = ToolRunner.run(conf, new AvgSix(), args);
System.exit(res);
}
[yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
-Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
thread "main" java.lang.IllegalArgumentException: The value of property
test must not be null at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
AvgSix.main(AvgSix.java:32) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881 ~]$
yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
/user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
java.lang.IllegalArgumentException: The value of property test must not be
null at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
AvgSix.main(AvgSix.java:32) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136)
On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> I think in your case it should be "*yarn jar
> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
> /user/yarn/input/samplefile.csv output*"
>
> + Naga
> ------------------------------
> *From:* Naganarasimha G R (Naga)
> *Sent:* Tuesday, October 06, 2015 20:16
> *To:* user@hadoop.apache.org
> *Subject:* RE: Passing Args to Mappers and Reducers
>
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> i think in your case it should be
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Tuesday, October 06, 2015 19:46
> *To:* user@hadoop.apache.org
> *Subject:* Re: Passing Args to Mappers and Reducers
>
> Hi Shahab,
> Thx for the quick reply.
> Yes, I am using the tool interface and can run the job successfully.
> I need to pass some extra args dynamically as part of the invocation.
>
> I have tried adding a space after the -D option and it did not work either
> (see below).
>
> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/sample.csv output
> Error: Could not find or load main class ista=666666
>
> It does not see to recognize the -D option.
>
> BR
>
> Ista
>
> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
> wrote:
>
>> Are you properly implementing the Tool interface?
>>
>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>
>> Also, there needs to be space between -D and the param name.
>>
>> Regards,
>> Shahab
>>
>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>> wrote:
>>
>>> Greetings to all,
>>> Is it possible to pass args to Mapper and Reducers via the command line
>>> args.
>>> I need to pass some args upon the MapRed job invocation so that I can
>>> pass these args via the Context object to the Mapper and Reducer code.
>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>> -D VM args but it did not succeed.
>>> Here are some failed code snippets:
>>>
>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>
>>> Please advise.
>>>
>>>
>>> Thx and BR
>>>
>>> Ista
>>>
>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Naga,
Thx for the follow up.
Still not working.
Here are the invocation errors and the source code.
[image: Inline image 1]
public class AvgSix extends Configured implements Tool{
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("test", System.getProperty("ista")); // <=== This is the line
where it is failing
int res = ToolRunner.run(conf, new AvgSix(), args);
System.exit(res);
}
[yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
-Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
thread "main" java.lang.IllegalArgumentException: The value of property
test must not be null at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
AvgSix.main(AvgSix.java:32) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881 ~]$
yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
/user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
java.lang.IllegalArgumentException: The value of property test must not be
null at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
AvgSix.main(AvgSix.java:32) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136)
On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> I think in your case it should be "*yarn jar
> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
> /user/yarn/input/samplefile.csv output*"
>
> + Naga
> ------------------------------
> *From:* Naganarasimha G R (Naga)
> *Sent:* Tuesday, October 06, 2015 20:16
> *To:* user@hadoop.apache.org
> *Subject:* RE: Passing Args to Mappers and Reducers
>
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> i think in your case it should be
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Tuesday, October 06, 2015 19:46
> *To:* user@hadoop.apache.org
> *Subject:* Re: Passing Args to Mappers and Reducers
>
> Hi Shahab,
> Thx for the quick reply.
> Yes, I am using the tool interface and can run the job successfully.
> I need to pass some extra args dynamically as part of the invocation.
>
> I have tried adding a space after the -D option and it did not work either
> (see below).
>
> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/sample.csv output
> Error: Could not find or load main class ista=666666
>
> It does not see to recognize the -D option.
>
> BR
>
> Ista
>
> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
> wrote:
>
>> Are you properly implementing the Tool interface?
>>
>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>
>> Also, there needs to be space between -D and the param name.
>>
>> Regards,
>> Shahab
>>
>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>> wrote:
>>
>>> Greetings to all,
>>> Is it possible to pass args to Mapper and Reducers via the command line
>>> args.
>>> I need to pass some args upon the MapRed job invocation so that I can
>>> pass these args via the Context object to the Mapper and Reducer code.
>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>> -D VM args but it did not succeed.
>>> Here are some failed code snippets:
>>>
>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>
>>> Please advise.
>>>
>>>
>>> Thx and BR
>>>
>>> Ista
>>>
>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Varun Saxena <vs...@gmail.com>.
Two configurations are used to pass command line opts to mappers and
reducers.
These are mapreduce.map.java.opts and mapreduce.reduce.java.opts for
mappers and reducers respectively.
There is a configuration named mapred.child.java.opts if you want same
arguments to be passed to both mappers and reducers.
You can also pass them via command line while submitting the job i.e.
*./yarn jar
../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
sleep -Dmapreduce.map.java.opts=<map opts> -m 10 -r 1 -mt 5000*
Varun.
On Tue, Oct 6, 2015 at 8:19 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> I think in your case it should be "*yarn jar
> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
> /user/yarn/input/samplefile.csv output*"
>
> + Naga
> ------------------------------
> *From:* Naganarasimha G R (Naga)
> *Sent:* Tuesday, October 06, 2015 20:16
> *To:* user@hadoop.apache.org
> *Subject:* RE: Passing Args to Mappers and Reducers
>
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> i think in your case it should be
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Tuesday, October 06, 2015 19:46
> *To:* user@hadoop.apache.org
> *Subject:* Re: Passing Args to Mappers and Reducers
>
> Hi Shahab,
> Thx for the quick reply.
> Yes, I am using the tool interface and can run the job successfully.
> I need to pass some extra args dynamically as part of the invocation.
>
> I have tried adding a space after the -D option and it did not work either
> (see below).
>
> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/sample.csv output
> Error: Could not find or load main class ista=666666
>
> It does not see to recognize the -D option.
>
> BR
>
> Ista
>
> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
> wrote:
>
>> Are you properly implementing the Tool interface?
>>
>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>
>> Also, there needs to be space between -D and the param name.
>>
>> Regards,
>> Shahab
>>
>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>> wrote:
>>
>>> Greetings to all,
>>> Is it possible to pass args to Mapper and Reducers via the command line
>>> args.
>>> I need to pass some args upon the MapRed job invocation so that I can
>>> pass these args via the Context object to the Mapper and Reducer code.
>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>> -D VM args but it did not succeed.
>>> Here are some failed code snippets:
>>>
>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>
>>> Please advise.
>>>
>>>
>>> Thx and BR
>>>
>>> Ista
>>>
>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Varun Saxena <vs...@gmail.com>.
Two configurations are used to pass command line opts to mappers and
reducers.
These are mapreduce.map.java.opts and mapreduce.reduce.java.opts for
mappers and reducers respectively.
There is a configuration named mapred.child.java.opts if you want same
arguments to be passed to both mappers and reducers.
You can also pass them via command line while submitting the job i.e.
*./yarn jar
../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
sleep -Dmapreduce.map.java.opts=<map opts> -m 10 -r 1 -mt 5000*
Varun.
On Tue, Oct 6, 2015 at 8:19 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> I think in your case it should be "*yarn jar
> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
> /user/yarn/input/samplefile.csv output*"
>
> + Naga
> ------------------------------
> *From:* Naganarasimha G R (Naga)
> *Sent:* Tuesday, October 06, 2015 20:16
> *To:* user@hadoop.apache.org
> *Subject:* RE: Passing Args to Mappers and Reducers
>
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> i think in your case it should be
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Tuesday, October 06, 2015 19:46
> *To:* user@hadoop.apache.org
> *Subject:* Re: Passing Args to Mappers and Reducers
>
> Hi Shahab,
> Thx for the quick reply.
> Yes, I am using the tool interface and can run the job successfully.
> I need to pass some extra args dynamically as part of the invocation.
>
> I have tried adding a space after the -D option and it did not work either
> (see below).
>
> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/sample.csv output
> Error: Could not find or load main class ista=666666
>
> It does not see to recognize the -D option.
>
> BR
>
> Ista
>
> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
> wrote:
>
>> Are you properly implementing the Tool interface?
>>
>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>
>> Also, there needs to be space between -D and the param name.
>>
>> Regards,
>> Shahab
>>
>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>> wrote:
>>
>>> Greetings to all,
>>> Is it possible to pass args to Mapper and Reducers via the command line
>>> args.
>>> I need to pass some args upon the MapRed job invocation so that I can
>>> pass these args via the Context object to the Mapper and Reducer code.
>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>> -D VM args but it did not succeed.
>>> Here are some failed code snippets:
>>>
>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>
>>> Please advise.
>>>
>>>
>>> Thx and BR
>>>
>>> Ista
>>>
>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Naga,
Thx for the follow up.
Still not working.
Here are the invocation errors and the source code.
[image: Inline image 1]
public class AvgSix extends Configured implements Tool{
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("test", System.getProperty("ista")); // <=== This is the line
where it is failing
int res = ToolRunner.run(conf, new AvgSix(), args);
System.exit(res);
}
[yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
-Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
thread "main" java.lang.IllegalArgumentException: The value of property
test must not be null at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
AvgSix.main(AvgSix.java:32) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881 ~]$
yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
/user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
java.lang.IllegalArgumentException: The value of property test must not be
null at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
AvgSix.main(AvgSix.java:32) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136)
On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> I think in your case it should be "*yarn jar
> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
> /user/yarn/input/samplefile.csv output*"
>
> + Naga
> ------------------------------
> *From:* Naganarasimha G R (Naga)
> *Sent:* Tuesday, October 06, 2015 20:16
> *To:* user@hadoop.apache.org
> *Subject:* RE: Passing Args to Mappers and Reducers
>
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> i think in your case it should be
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Tuesday, October 06, 2015 19:46
> *To:* user@hadoop.apache.org
> *Subject:* Re: Passing Args to Mappers and Reducers
>
> Hi Shahab,
> Thx for the quick reply.
> Yes, I am using the tool interface and can run the job successfully.
> I need to pass some extra args dynamically as part of the invocation.
>
> I have tried adding a space after the -D option and it did not work either
> (see below).
>
> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/sample.csv output
> Error: Could not find or load main class ista=666666
>
> It does not see to recognize the -D option.
>
> BR
>
> Ista
>
> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
> wrote:
>
>> Are you properly implementing the Tool interface?
>>
>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>
>> Also, there needs to be space between -D and the param name.
>>
>> Regards,
>> Shahab
>>
>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>> wrote:
>>
>>> Greetings to all,
>>> Is it possible to pass args to Mapper and Reducers via the command line
>>> args.
>>> I need to pass some args upon the MapRed job invocation so that I can
>>> pass these args via the Context object to the Mapper and Reducer code.
>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>> -D VM args but it did not succeed.
>>> Here are some failed code snippets:
>>>
>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>
>>> Please advise.
>>>
>>>
>>> Thx and BR
>>>
>>> Ista
>>>
>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Varun Saxena <vs...@gmail.com>.
Two configurations are used to pass command line opts to mappers and
reducers.
These are mapreduce.map.java.opts and mapreduce.reduce.java.opts for
mappers and reducers respectively.
There is a configuration named mapred.child.java.opts if you want same
arguments to be passed to both mappers and reducers.
You can also pass them via command line while submitting the job i.e.
*./yarn jar
../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
sleep -Dmapreduce.map.java.opts=<map opts> -m 10 -r 1 -mt 5000*
Varun.
On Tue, Oct 6, 2015 at 8:19 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> I think in your case it should be "*yarn jar
> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
> /user/yarn/input/samplefile.csv output*"
>
> + Naga
> ------------------------------
> *From:* Naganarasimha G R (Naga)
> *Sent:* Tuesday, October 06, 2015 20:16
> *To:* user@hadoop.apache.org
> *Subject:* RE: Passing Args to Mappers and Reducers
>
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> i think in your case it should be
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Tuesday, October 06, 2015 19:46
> *To:* user@hadoop.apache.org
> *Subject:* Re: Passing Args to Mappers and Reducers
>
> Hi Shahab,
> Thx for the quick reply.
> Yes, I am using the tool interface and can run the job successfully.
> I need to pass some extra args dynamically as part of the invocation.
>
> I have tried adding a space after the -D option and it did not work either
> (see below).
>
> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/sample.csv output
> Error: Could not find or load main class ista=666666
>
> It does not see to recognize the -D option.
>
> BR
>
> Ista
>
> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
> wrote:
>
>> Are you properly implementing the Tool interface?
>>
>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>
>> Also, there needs to be space between -D and the param name.
>>
>> Regards,
>> Shahab
>>
>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>> wrote:
>>
>>> Greetings to all,
>>> Is it possible to pass args to Mapper and Reducers via the command line
>>> args.
>>> I need to pass some args upon the MapRed job invocation so that I can
>>> pass these args via the Context object to the Mapper and Reducer code.
>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>> -D VM args but it did not succeed.
>>> Here are some failed code snippets:
>>>
>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>
>>> Please advise.
>>>
>>>
>>> Thx and BR
>>>
>>> Ista
>>>
>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Varun Saxena <vs...@gmail.com>.
Two configurations are used to pass command line opts to mappers and
reducers.
These are mapreduce.map.java.opts and mapreduce.reduce.java.opts for
mappers and reducers respectively.
There is a configuration named mapred.child.java.opts if you want same
arguments to be passed to both mappers and reducers.
You can also pass them via command line while submitting the job i.e.
*./yarn jar
../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
sleep -Dmapreduce.map.java.opts=<map opts> -m 10 -r 1 -mt 5000*
Varun.
On Tue, Oct 6, 2015 at 8:19 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> I think in your case it should be "*yarn jar
> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
> /user/yarn/input/samplefile.csv output*"
>
> + Naga
> ------------------------------
> *From:* Naganarasimha G R (Naga)
> *Sent:* Tuesday, October 06, 2015 20:16
> *To:* user@hadoop.apache.org
> *Subject:* RE: Passing Args to Mappers and Reducers
>
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> i think in your case it should be
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Tuesday, October 06, 2015 19:46
> *To:* user@hadoop.apache.org
> *Subject:* Re: Passing Args to Mappers and Reducers
>
> Hi Shahab,
> Thx for the quick reply.
> Yes, I am using the tool interface and can run the job successfully.
> I need to pass some extra args dynamically as part of the invocation.
>
> I have tried adding a space after the -D option and it did not work either
> (see below).
>
> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/sample.csv output
> Error: Could not find or load main class ista=666666
>
> It does not see to recognize the -D option.
>
> BR
>
> Ista
>
> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
> wrote:
>
>> Are you properly implementing the Tool interface?
>>
>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>
>> Also, there needs to be space between -D and the param name.
>>
>> Regards,
>> Shahab
>>
>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>> wrote:
>>
>>> Greetings to all,
>>> Is it possible to pass args to Mapper and Reducers via the command line
>>> args.
>>> I need to pass some args upon the MapRed job invocation so that I can
>>> pass these args via the Context object to the Mapper and Reducer code.
>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>> -D VM args but it did not succeed.
>>> Here are some failed code snippets:
>>>
>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>
>>> Please advise.
>>>
>>>
>>> Thx and BR
>>>
>>> Ista
>>>
>>
>>
>
Re: Passing Args to Mappers and Reducers
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Naga,
Thx for the follow up.
Still not working.
Here are the invocation errors and the source code.
[image: Inline image 1]
public class AvgSix extends Configured implements Tool{
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("test", System.getProperty("ista")); // <=== This is the line
where it is failing
int res = ToolRunner.run(conf, new AvgSix(), args);
System.exit(res);
}
[yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
-Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
thread "main" java.lang.IllegalArgumentException: The value of property
test must not be null at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
AvgSix.main(AvgSix.java:32) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881 ~]$
yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
/user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
java.lang.IllegalArgumentException: The value of property test must not be
null at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
AvgSix.main(AvgSix.java:32) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136)
On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> I think in your case it should be "*yarn jar
> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
> /user/yarn/input/samplefile.csv output*"
>
> + Naga
> ------------------------------
> *From:* Naganarasimha G R (Naga)
> *Sent:* Tuesday, October 06, 2015 20:16
> *To:* user@hadoop.apache.org
> *Subject:* RE: Passing Args to Mappers and Reducers
>
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> i think in your case it should be
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Tuesday, October 06, 2015 19:46
> *To:* user@hadoop.apache.org
> *Subject:* Re: Passing Args to Mappers and Reducers
>
> Hi Shahab,
> Thx for the quick reply.
> Yes, I am using the tool interface and can run the job successfully.
> I need to pass some extra args dynamically as part of the invocation.
>
> I have tried adding a space after the -D option and it did not work either
> (see below).
>
> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/sample.csv output
> Error: Could not find or load main class ista=666666
>
> It does not see to recognize the -D option.
>
> BR
>
> Ista
>
> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>
> wrote:
>
>> Are you properly implementing the Tool interface?
>>
>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>
>> Also, there needs to be space between -D and the param name.
>>
>> Regards,
>> Shahab
>>
>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
>> wrote:
>>
>>> Greetings to all,
>>> Is it possible to pass args to Mapper and Reducers via the command line
>>> args.
>>> I need to pass some args upon the MapRed job invocation so that I can
>>> pass these args via the Context object to the Mapper and Reducer code.
>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>> -D VM args but it did not succeed.
>>> Here are some failed code snippets:
>>>
>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>>
>>> Please advise.
>>>
>>>
>>> Thx and BR
>>>
>>> Ista
>>>
>>
>>
>
RE: Passing Args to Mappers and Reducers
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Ista,
In general we need to give after specifying the class name like
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000
I think in your case it should be "yarn jar /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666" /user/yarn/input/samplefile.csv output"
+ Naga
________________________________
From: Naganarasimha G R (Naga)
Sent: Tuesday, October 06, 2015 20:16
To: user@hadoop.apache.org
Subject: RE: Passing Args to Mappers and Reducers
Hi Ista,
In general we need to give after specifying the class name like
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000
i think in your case it should be
________________________________
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Tuesday, October 06, 2015 19:46
To: user@hadoop.apache.org
Subject: Re: Passing Args to Mappers and Reducers
Hi Shahab,
Thx for the quick reply.
Yes, I am using the tool interface and can run the job successfully.
I need to pass some extra args dynamically as part of the invocation.
I have tried adding a space after the -D option and it did not work either (see below).
~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/sample.csv output
Error: Could not find or load main class ista=666666
It does not see to recognize the -D option.
BR
Ista
On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>> wrote:
Are you properly implementing the Tool interface?
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
Also, there needs to be space between -D and the param name.
Regards,
Shahab
On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>> wrote:
Greetings to all,
Is it possible to pass args to Mapper and Reducers via the command line args.
I need to pass some args upon the MapRed job invocation so that I can pass these args via the Context object to the Mapper and Reducer code.
I am currently running Hadoop2.7 and tried to pass some args using the -D VM args but it did not succeed.
Here are some failed code snippets:
~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output -Dista="666666"
Please advise.
Thx and BR
Ista
RE: Passing Args to Mappers and Reducers
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Ista,
In general we need to give after specifying the class name like
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000
I think in your case it should be "yarn jar /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666" /user/yarn/input/samplefile.csv output"
+ Naga
________________________________
From: Naganarasimha G R (Naga)
Sent: Tuesday, October 06, 2015 20:16
To: user@hadoop.apache.org
Subject: RE: Passing Args to Mappers and Reducers
Hi Ista,
In general we need to give after specifying the class name like
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000
i think in your case it should be
________________________________
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Tuesday, October 06, 2015 19:46
To: user@hadoop.apache.org
Subject: Re: Passing Args to Mappers and Reducers
Hi Shahab,
Thx for the quick reply.
Yes, I am using the tool interface and can run the job successfully.
I need to pass some extra args dynamically as part of the invocation.
I have tried adding a space after the -D option and it did not work either (see below).
~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/sample.csv output
Error: Could not find or load main class ista=666666
It does not see to recognize the -D option.
BR
Ista
On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>> wrote:
Are you properly implementing the Tool interface?
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
Also, there needs to be space between -D and the param name.
Regards,
Shahab
On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>> wrote:
Greetings to all,
Is it possible to pass args to Mapper and Reducers via the command line args.
I need to pass some args upon the MapRed job invocation so that I can pass these args via the Context object to the Mapper and Reducer code.
I am currently running Hadoop2.7 and tried to pass some args using the -D VM args but it did not succeed.
Here are some failed code snippets:
~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output -Dista="666666"
Please advise.
Thx and BR
Ista
RE: Passing Args to Mappers and Reducers
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Ista,
In general we need to give after specifying the class name like
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000
I think in your case it should be "yarn jar /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666" /user/yarn/input/samplefile.csv output"
+ Naga
________________________________
From: Naganarasimha G R (Naga)
Sent: Tuesday, October 06, 2015 20:16
To: user@hadoop.apache.org
Subject: RE: Passing Args to Mappers and Reducers
Hi Ista,
In general we need to give after specifying the class name like
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000
i think in your case it should be
________________________________
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Tuesday, October 06, 2015 19:46
To: user@hadoop.apache.org
Subject: Re: Passing Args to Mappers and Reducers
Hi Shahab,
Thx for the quick reply.
Yes, I am using the tool interface and can run the job successfully.
I need to pass some extra args dynamically as part of the invocation.
I have tried adding a space after the -D option and it did not work either (see below).
~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/sample.csv output
Error: Could not find or load main class ista=666666
It does not see to recognize the -D option.
BR
Ista
On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>> wrote:
Are you properly implementing the Tool interface?
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
Also, there needs to be space between -D and the param name.
Regards,
Shahab
On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>> wrote:
Greetings to all,
Is it possible to pass args to Mapper and Reducers via the command line args.
I need to pass some args upon the MapRed job invocation so that I can pass these args via the Context object to the Mapper and Reducer code.
I am currently running Hadoop2.7 and tried to pass some args using the -D VM args but it did not succeed.
Here are some failed code snippets:
~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output -Dista="666666"
Please advise.
Thx and BR
Ista
RE: Passing Args to Mappers and Reducers
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Ista,
In general we need to give after specifying the class name like
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000
I think in your case it should be "yarn jar /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666" /user/yarn/input/samplefile.csv output"
+ Naga
________________________________
From: Naganarasimha G R (Naga)
Sent: Tuesday, October 06, 2015 20:16
To: user@hadoop.apache.org
Subject: RE: Passing Args to Mappers and Reducers
Hi Ista,
In general we need to give after specifying the class name like
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000
i think in your case it should be
________________________________
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Tuesday, October 06, 2015 19:46
To: user@hadoop.apache.org
Subject: Re: Passing Args to Mappers and Reducers
Hi Shahab,
Thx for the quick reply.
Yes, I am using the tool interface and can run the job successfully.
I need to pass some extra args dynamically as part of the invocation.
I have tried adding a space after the -D option and it did not work either (see below).
~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/sample.csv output
Error: Could not find or load main class ista=666666
It does not see to recognize the -D option.
BR
Ista
On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>> wrote:
Are you properly implementing the Tool interface?
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
Also, there needs to be space between -D and the param name.
Regards,
Shahab
On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>> wrote:
Greetings to all,
Is it possible to pass args to Mapper and Reducers via the command line args.
I need to pass some args upon the MapRed job invocation so that I can pass these args via the Context object to the Mapper and Reducer code.
I am currently running Hadoop2.7 and tried to pass some args using the -D VM args but it did not succeed.
Here are some failed code snippets:
~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output -Dista="666666"
Please advise.
Thx and BR
Ista
RE: Passing Args to Mappers and Reducers
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Ista,
In general we need to give after specifying the class name like
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000
i think in your case it should be
________________________________
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Tuesday, October 06, 2015 19:46
To: user@hadoop.apache.org
Subject: Re: Passing Args to Mappers and Reducers
Hi Shahab,
Thx for the quick reply.
Yes, I am using the tool interface and can run the job successfully.
I need to pass some extra args dynamically as part of the invocation.
I have tried adding a space after the -D option and it did not work either (see below).
~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/sample.csv output
Error: Could not find or load main class ista=666666
It does not see to recognize the -D option.
BR
Ista
On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>> wrote:
Are you properly implementing the Tool interface?
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
Also, there needs to be space between -D and the param name.
Regards,
Shahab
On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>> wrote:
Greetings to all,
Is it possible to pass args to Mapper and Reducers via the command line args.
I need to pass some args upon the MapRed job invocation so that I can pass these args via the Context object to the Mapper and Reducer code.
I am currently running Hadoop2.7 and tried to pass some args using the -D VM args but it did not succeed.
Here are some failed code snippets:
~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output -Dista="666666"
Please advise.
Thx and BR
Ista
RE: Passing Args to Mappers and Reducers
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Ista,
In general we need to give after specifying the class name like
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000
i think in your case it should be
________________________________
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Tuesday, October 06, 2015 19:46
To: user@hadoop.apache.org
Subject: Re: Passing Args to Mappers and Reducers
Hi Shahab,
Thx for the quick reply.
Yes, I am using the tool interface and can run the job successfully.
I need to pass some extra args dynamically as part of the invocation.
I have tried adding a space after the -D option and it did not work either (see below).
~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/sample.csv output
Error: Could not find or load main class ista=666666
It does not see to recognize the -D option.
BR
Ista
On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>> wrote:
Are you properly implementing the Tool interface?
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
Also, there needs to be space between -D and the param name.
Regards,
Shahab
On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>> wrote:
Greetings to all,
Is it possible to pass args to Mapper and Reducers via the command line args.
I need to pass some args upon the MapRed job invocation so that I can pass these args via the Context object to the Mapper and Reducer code.
I am currently running Hadoop2.7 and tried to pass some args using the -D VM args but it did not succeed.
Here are some failed code snippets:
~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output -Dista="666666"
Please advise.
Thx and BR
Ista
RE: Passing Args to Mappers and Reducers
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Ista,
In general we need to give after specifying the class name like
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000
i think in your case it should be
________________________________
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Tuesday, October 06, 2015 19:46
To: user@hadoop.apache.org
Subject: Re: Passing Args to Mappers and Reducers
Hi Shahab,
Thx for the quick reply.
Yes, I am using the tool interface and can run the job successfully.
I need to pass some extra args dynamically as part of the invocation.
I have tried adding a space after the -D option and it did not work either (see below).
~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/sample.csv output
Error: Could not find or load main class ista=666666
It does not see to recognize the -D option.
BR
Ista
On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>> wrote:
Are you properly implementing the Tool interface?
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
Also, there needs to be space between -D and the param name.
Regards,
Shahab
On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>> wrote:
Greetings to all,
Is it possible to pass args to Mapper and Reducers via the command line args.
I need to pass some args upon the MapRed job invocation so that I can pass these args via the Context object to the Mapper and Reducer code.
I am currently running Hadoop2.7 and tried to pass some args using the -D VM args but it did not succeed.
Here are some failed code snippets:
~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output -Dista="666666"
Please advise.
Thx and BR
Ista
RE: Passing Args to Mappers and Reducers
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Ista,
In general we need to give after specifying the class name like
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000
i think in your case it should be
________________________________
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Tuesday, October 06, 2015 19:46
To: user@hadoop.apache.org
Subject: Re: Passing Args to Mappers and Reducers
Hi Shahab,
Thx for the quick reply.
Yes, I am using the tool interface and can run the job successfully.
I need to pass some extra args dynamically as part of the invocation.
I have tried adding a space after the -D option and it did not work either (see below).
~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/sample.csv output
Error: Could not find or load main class ista=666666
It does not see to recognize the -D option.
BR
Ista
On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com>> wrote:
Are you properly implementing the Tool interface?
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
Also, there needs to be space between -D and the param name.
Regards,
Shahab
On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>> wrote:
Greetings to all,
Is it possible to pass args to Mapper and Reducers via the command line args.
I need to pass some args upon the MapRed job invocation so that I can pass these args via the Context object to the Mapper and Reducer code.
I am currently running Hadoop2.7 and tried to pass some args using the -D VM args but it did not succeed.
Here are some failed code snippets:
~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output
~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar /user/yarn/input/samplefile.csv output -Dista="666666"
Please advise.
Thx and BR
Ista
Re: Passing Args to Mappers and Reducers
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Shahab,
Thx for the quick reply.
Yes, I am using the tool interface and can run the job successfully.
I need to pass some extra args dynamically as part of the invocation.
I have tried adding a space after the -D option and it did not work either
(see below).
~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
/user/yarn/input/sample.csv output
Error: Could not find or load main class ista=666666
It does not see to recognize the -D option.
BR
Ista
On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com> wrote:
> Are you properly implementing the Tool interface?
>
> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>
> Also, there needs to be space between -D and the param name.
>
> Regards,
> Shahab
>
> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
> wrote:
>
>> Greetings to all,
>> Is it possible to pass args to Mapper and Reducers via the command line
>> args.
>> I need to pass some args upon the MapRed job invocation so that I can
>> pass these args via the Context object to the Mapper and Reducer code.
>> I am currently running Hadoop2.7 and tried to pass some args using the -D
>> VM args but it did not succeed.
>> Here are some failed code snippets:
>>
>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/samplefile.csv output
>>
>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/samplefile.csv output
>>
>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>
>> Please advise.
>>
>>
>> Thx and BR
>>
>> Ista
>>
>
>
Re: Passing Args to Mappers and Reducers
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Shahab,
Thx for the quick reply.
Yes, I am using the tool interface and can run the job successfully.
I need to pass some extra args dynamically as part of the invocation.
I have tried adding a space after the -D option and it did not work either
(see below).
~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
/user/yarn/input/sample.csv output
Error: Could not find or load main class ista=666666
It does not see to recognize the -D option.
BR
Ista
On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com> wrote:
> Are you properly implementing the Tool interface?
>
> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>
> Also, there needs to be space between -D and the param name.
>
> Regards,
> Shahab
>
> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
> wrote:
>
>> Greetings to all,
>> Is it possible to pass args to Mapper and Reducers via the command line
>> args.
>> I need to pass some args upon the MapRed job invocation so that I can
>> pass these args via the Context object to the Mapper and Reducer code.
>> I am currently running Hadoop2.7 and tried to pass some args using the -D
>> VM args but it did not succeed.
>> Here are some failed code snippets:
>>
>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/samplefile.csv output
>>
>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/samplefile.csv output
>>
>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>
>> Please advise.
>>
>>
>> Thx and BR
>>
>> Ista
>>
>
>
Re: Passing Args to Mappers and Reducers
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Shahab,
Thx for the quick reply.
Yes, I am using the tool interface and can run the job successfully.
I need to pass some extra args dynamically as part of the invocation.
I have tried adding a space after the -D option and it did not work either
(see below).
~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
/user/yarn/input/sample.csv output
Error: Could not find or load main class ista=666666
It does not see to recognize the -D option.
BR
Ista
On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com> wrote:
> Are you properly implementing the Tool interface?
>
> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>
> Also, there needs to be space between -D and the param name.
>
> Regards,
> Shahab
>
> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
> wrote:
>
>> Greetings to all,
>> Is it possible to pass args to Mapper and Reducers via the command line
>> args.
>> I need to pass some args upon the MapRed job invocation so that I can
>> pass these args via the Context object to the Mapper and Reducer code.
>> I am currently running Hadoop2.7 and tried to pass some args using the -D
>> VM args but it did not succeed.
>> Here are some failed code snippets:
>>
>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/samplefile.csv output
>>
>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/samplefile.csv output
>>
>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>
>> Please advise.
>>
>>
>> Thx and BR
>>
>> Ista
>>
>
>
Re: Passing Args to Mappers and Reducers
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Shahab,
Thx for the quick reply.
Yes, I am using the tool interface and can run the job successfully.
I need to pass some extra args dynamically as part of the invocation.
I have tried adding a space after the -D option and it did not work either
(see below).
~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
/user/yarn/input/sample.csv output
Error: Could not find or load main class ista=666666
It does not see to recognize the -D option.
BR
Ista
On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <sh...@gmail.com> wrote:
> Are you properly implementing the Tool interface?
>
> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>
> Also, there needs to be space between -D and the param name.
>
> Regards,
> Shahab
>
> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
> wrote:
>
>> Greetings to all,
>> Is it possible to pass args to Mapper and Reducers via the command line
>> args.
>> I need to pass some args upon the MapRed job invocation so that I can
>> pass these args via the Context object to the Mapper and Reducer code.
>> I am currently running Hadoop2.7 and tried to pass some args using the -D
>> VM args but it did not succeed.
>> Here are some failed code snippets:
>>
>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/samplefile.csv output
>>
>> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/samplefile.csv output
>>
>> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
>> /user/yarn/input/samplefile.csv output -Dista="666666"
>>
>> Please advise.
>>
>>
>> Thx and BR
>>
>> Ista
>>
>
>
Re: Passing Args to Mappers and Reducers
Posted by Shahab Yunus <sh...@gmail.com>.
Are you properly implementing the Tool interface?
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
Also, there needs to be space between -D and the param name.
Regards,
Shahab
On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Greetings to all,
> Is it possible to pass args to Mapper and Reducers via the command line
> args.
> I need to pass some args upon the MapRed job invocation so that I can pass
> these args via the Context object to the Mapper and Reducer code.
> I am currently running Hadoop2.7 and tried to pass some args using the -D
> VM args but it did not succeed.
> Here are some failed code snippets:
>
> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/samplefile.csv output
>
> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/samplefile.csv output
>
> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/samplefile.csv output -Dista="666666"
>
> Please advise.
>
>
> Thx and BR
>
> Ista
>
Re: Passing Args to Mappers and Reducers
Posted by Shahab Yunus <sh...@gmail.com>.
Are you properly implementing the Tool interface?
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
Also, there needs to be space between -D and the param name.
Regards,
Shahab
On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Greetings to all,
> Is it possible to pass args to Mapper and Reducers via the command line
> args.
> I need to pass some args upon the MapRed job invocation so that I can pass
> these args via the Context object to the Mapper and Reducer code.
> I am currently running Hadoop2.7 and tried to pass some args using the -D
> VM args but it did not succeed.
> Here are some failed code snippets:
>
> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/samplefile.csv output
>
> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/samplefile.csv output
>
> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/samplefile.csv output -Dista="666666"
>
> Please advise.
>
>
> Thx and BR
>
> Ista
>
Re: Passing Args to Mappers and Reducers
Posted by Shahab Yunus <sh...@gmail.com>.
Are you properly implementing the Tool interface?
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
Also, there needs to be space between -D and the param name.
Regards,
Shahab
On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Greetings to all,
> Is it possible to pass args to Mapper and Reducers via the command line
> args.
> I need to pass some args upon the MapRed job invocation so that I can pass
> these args via the Context object to the Mapper and Reducer code.
> I am currently running Hadoop2.7 and tried to pass some args using the -D
> VM args but it did not succeed.
> Here are some failed code snippets:
>
> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/samplefile.csv output
>
> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/samplefile.csv output
>
> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/samplefile.csv output -Dista="666666"
>
> Please advise.
>
>
> Thx and BR
>
> Ista
>
Re: Passing Args to Mappers and Reducers
Posted by Shahab Yunus <sh...@gmail.com>.
Are you properly implementing the Tool interface?
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
Also, there needs to be space between -D and the param name.
Regards,
Shahab
On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Greetings to all,
> Is it possible to pass args to Mapper and Reducers via the command line
> args.
> I need to pass some args upon the MapRed job invocation so that I can pass
> these args via the Context object to the Mapper and Reducer code.
> I am currently running Hadoop2.7 and tried to pass some args using the -D
> VM args but it did not succeed.
> Here are some failed code snippets:
>
> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/samplefile.csv output
>
> ~]$ yarn jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/samplefile.csv output
>
> ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/samplefile.csv output -Dista="666666"
>
> Please advise.
>
>
> Thx and BR
>
> Ista
>