You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Neil Yalowitz <ne...@gmail.com> on 2012/01/24 21:14:52 UTC
passing an arbitrary property to a job with map-reduce
Hi all,
I'm attempting to pass a property to an MR job with an oozie workflow
structured with a <map-reduce> tag. This was trivial with a <java> tag, as
I would call my own main(String[] args) and instantiate my own Driver and
then ToolRunner.run(). Because I was coding my own main(), I had an
opportunity to parse/set properties easily:
workflow.xml
<java>
<main-class>com.acme.MyMain</main-class>
<configuration>
...
</configuration>
<arg>foovalue</arg>
<arg>barvalue</arg>
</java>
...and capture those arguments:
MyMain.java
public static void main(String[] args) throws Exception {
...
String foo = args[0];
String bar = args[1];
String[] argsAll = {foo, bar};
Configuration conf = new Configuration();
ToolRunner.run(conf, new MyDriver(), argsAll);
}
This worked fine, but I can't figure out how to capture those properties
using a <map-reduce> tag. I attempt something like this:
workflow.xml
<map-reduce>
...
<configuration>
<property>
<name>foo</name>
<value>foovalue</value>
</property>
</configuration>
</map-reduce>
However, the foo property is not set in the created
org.apache.hadoop.conf.Configuration object:
MyMapper.java
...
@Override
protected void setup(Context context)... {
context.getConfiguration().get("foo"); //null!
}
Is there a way to pass an arbitrary property (ie-- not a mapred.* property)
to a job with an Oozie workflow using a <map-reduce> tag? Perhaps I am
missing something obvious...
Thanks!
Neil Yalowitz
Re: passing an arbitrary property to a job with map-reduce
Posted by Neil Yalowitz <ne...@gmail.com>.
Ah yes, I see now... we are using the new API in our Mapper class.
import org.apache.hadoop.mapreduce.Mapper;
Neil
On Wed, Jan 25, 2012 at 1:45 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Neil,
>
> It looks you are using the new MR API for your mapper class.
>
> I'd suggest start from the Oozie MR example working and modified it to
> verify that property is propagated.
>
> Thanks.
>
> Alejadnro
>
> On Wed, Jan 25, 2012 at 10:12 AM, Neil Yalowitz <ne...@gmail.com>
> wrote:
> > Forgot to mention: I'm testing with Oozie v3.0.0 (but our production
> system
> > is actually v3.0.2).
> >
> > Thanks,
> > Neil
> >
> > On Wed, Jan 25, 2012 at 12:55 PM, Neil Yalowitz <neilyalowitz@gmail.com
> >wrote:
> >
> >> Yes, see these Pastebin links:
> >>
> >>
> >> workflow.xml
> >> http://pastebin.com/MqXwecjQ
> >>
> >> ACMEMapper.java
> >> http://pastebin.com/Y9rFcU97
> >>
> >>
> >> Thanks!
> >> Neil
> >>
> >> On Wed, Jan 25, 2012 at 12:23 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >>
> >>> Neil,
> >>>
> >>> You may be missing something. This should work.
> >>>
> >>> would you mind sharing a minimal WF.xml and MR java that shows your
> >>> problem?
> >>>
> >>> Thanks.
> >>>
> >>> Alejandro
> >>>
> >>> On Wed, Jan 25, 2012 at 9:03 AM, Neil Yalowitz <neilyalowitz@gmail.com
> >
> >>> wrote:
> >>> > Alejandro,
> >>> >
> >>> > Yes, we were using the new MR API, so I modified the workflow.xml to
> >>> remove
> >>> > mapred.mapper.new-api and re-tested. I still cannot pass a property
> to
> >>> my
> >>> > MR job.
> >>> >
> >>> > Is passing a non mapred.* property to a <map-reduce> job supported
> with
> >>> > Oozie? Or must I use the <java> job and parse the property in my
> >>> main()?
> >>> >
> >>> >
> >>> > Thanks,
> >>> > Neil
> >>> >
> >>> > On Tue, Jan 24, 2012 at 6:00 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> >>> >wrote:
> >>> >
> >>> >> Neil,
> >>> >>
> >>> >> Are you trying to use the new MR API?
> >>> >>
> >>> >> If so please look at
> >>> >>
> >>> >> https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
> >>> >>
> >>> >> The "How to run Map-reduce job written using new Hadoop API?"
> section.
> >>> >>
> >>> >> Thanks.
> >>> >>
> >>> >> Alejandro
> >>> >>
> >>> >> On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <
> >>> neilyalowitz@gmail.com>
> >>> >> wrote:
> >>> >> >
> >>> >> > Hi all,
> >>> >> >
> >>> >> > I'm attempting to pass a property to an MR job with an oozie
> workflow
> >>> >> > structured with a <map-reduce> tag. This was trivial with a
> <java>
> >>> tag,
> >>> >> as
> >>> >> > I would call my own main(String[] args) and instantiate my own
> >>> Driver and
> >>> >> > then ToolRunner.run(). Because I was coding my own main(), I had
> an
> >>> >> > opportunity to parse/set properties easily:
> >>> >> >
> >>> >> > workflow.xml
> >>> >> > <java>
> >>> >> > <main-class>com.acme.MyMain</main-class>
> >>> >> > <configuration>
> >>> >> > ...
> >>> >> > </configuration>
> >>> >> > <arg>foovalue</arg>
> >>> >> > <arg>barvalue</arg>
> >>> >> > </java>
> >>> >> >
> >>> >> > ...and capture those arguments:
> >>> >> >
> >>> >> > MyMain.java
> >>> >> > public static void main(String[] args) throws Exception {
> >>> >> > ...
> >>> >> > String foo = args[0];
> >>> >> > String bar = args[1];
> >>> >> > String[] argsAll = {foo, bar};
> >>> >> > Configuration conf = new Configuration();
> >>> >> > ToolRunner.run(conf, new MyDriver(), argsAll);
> >>> >> > }
> >>> >> >
> >>> >> > This worked fine, but I can't figure out how to capture those
> >>> properties
> >>> >> > using a <map-reduce> tag. I attempt something like this:
> >>> >> >
> >>> >> > workflow.xml
> >>> >> > <map-reduce>
> >>> >> > ...
> >>> >> > <configuration>
> >>> >> > <property>
> >>> >> > <name>foo</name>
> >>> >> > <value>foovalue</value>
> >>> >> > </property>
> >>> >> > </configuration>
> >>> >> > </map-reduce>
> >>> >> >
> >>> >> > However, the foo property is not set in the created
> >>> >> > org.apache.hadoop.conf.Configuration object:
> >>> >> >
> >>> >> > MyMapper.java
> >>> >> > ...
> >>> >> > @Override
> >>> >> > protected void setup(Context context)... {
> >>> >> > context.getConfiguration().get("foo"); //null!
> >>> >> > }
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > Is there a way to pass an arbitrary property (ie-- not a mapred.*
> >>> >> property)
> >>> >> > to a job with an Oozie workflow using a <map-reduce> tag?
> Perhaps I
> >>> am
> >>> >> > missing something obvious...
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > Thanks!
> >>> >> >
> >>> >> >
> >>> >> > Neil Yalowitz
> >>> >>
> >>>
> >>
> >>
>
Re: passing an arbitrary property to a job with map-reduce
Posted by Neil Yalowitz <ne...@gmail.com>.
I just wanted to confirm: this works with the new API as well using 3.0.0
or 3.0.2.
It previous was not working because I made a deployment mistake and was
running against a stale JAR. Total human error on my part and not an issue
with Oozie.
Thanks for everyone's assistance, however. I appreciate the willingness to
help.
Neil Yalowitz
On Wed, Jan 25, 2012 at 2:29 PM, Neil Yalowitz <ne...@gmail.com>wrote:
> Ok, good to know. Alejandro, Mohammad, Virag thanks for the help.
>
>
> Neil Yalowitz
>
>
> On Wed, Jan 25, 2012 at 2:18 PM, Virag Kothari <vi...@yahoo-inc.com>wrote:
>
>> Hey Neil,
>>
>> If you using the old API, you have to override the configure() of Mapper.
>>
>> E.g
>>
>> import org.apache.hadoop.mapred.Mapper;
>>
>> public class SampleMapper implements Mapper<LongWritable, Text,
>> LongWritable, Text> {
>>
>> public void configure(JobConf jobConf) {
>> System.out.println(jobConf.get("foo")); //should work
>> }
>> }
>>
>> Thanks,
>> Virag
>>
>>
>> On 1/25/12 11:09 AM, "Mohammad Islam" <mi...@yahoo.com> wrote:
>>
>> > One of our dev(API) has successfully tested this case with old API.
>> > I would ask Virag to share his findings/usage preferably as code
>> snippet.
>> >
>> > Regards,
>> > Mohammad
>> >
>> >
>> > ----- Original Message -----
>> > From: Alejandro Abdelnur <tu...@cloudera.com>
>> > To: oozie-users@incubator.apache.org
>> > Cc:
>> > Sent: Wednesday, January 25, 2012 10:45 AM
>> > Subject: Re: passing an arbitrary property to a job with map-reduce
>> >
>> > Neil,
>> >
>> > It looks you are using the new MR API for your mapper class.
>> >
>> > I'd suggest start from the Oozie MR example working and modified it to
>> > verify that property is propagated.
>> >
>> > Thanks.
>> >
>> > Alejadnro
>> >
>> > On Wed, Jan 25, 2012 at 10:12 AM, Neil Yalowitz <neilyalowitz@gmail.com
>> >
>> > wrote:
>> >> Forgot to mention: I'm testing with Oozie v3.0.0 (but our production
>> system
>> >> is actually v3.0.2).
>> >>
>> >> Thanks,
>> >> Neil
>> >>
>> >> On Wed, Jan 25, 2012 at 12:55 PM, Neil Yalowitz
>> >> <ne...@gmail.com>wrote:
>> >>
>> >>> Yes, see these Pastebin links:
>> >>>
>> >>>
>> >>> workflow.xml
>> >>> http://pastebin.com/MqXwecjQ
>> >>>
>> >>> ACMEMapper.java
>> >>> http://pastebin.com/Y9rFcU97
>> >>>
>> >>>
>> >>> Thanks!
>> >>> Neil
>> >>>
>> >>> On Wed, Jan 25, 2012 at 12:23 PM, Alejandro Abdelnur
>> >>> <tu...@cloudera.com>wrote:
>> >>>
>> >>>> Neil,
>> >>>>
>> >>>> You may be missing something. This should work.
>> >>>>
>> >>>> would you mind sharing a minimal WF.xml and MR java that shows your
>> >>>> problem?
>> >>>>
>> >>>> Thanks.
>> >>>>
>> >>>> Alejandro
>> >>>>
>> >>>> On Wed, Jan 25, 2012 at 9:03 AM, Neil Yalowitz <
>> neilyalowitz@gmail.com>
>> >>>> wrote:
>> >>>>> Alejandro,
>> >>>>>
>> >>>>> Yes, we were using the new MR API, so I modified the workflow.xml to
>> >>>> remove
>> >>>>> mapred.mapper.new-api and re-tested. I still cannot pass a
>> property to
>> >>>> my
>> >>>>> MR job.
>> >>>>>
>> >>>>> Is passing a non mapred.* property to a <map-reduce> job supported
>> with
>> >>>>> Oozie? Or must I use the <java> job and parse the property in my
>> >>>> main()?
>> >>>>>
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Neil
>> >>>>>
>> >>>>> On Tue, Jan 24, 2012 at 6:00 PM, Alejandro Abdelnur <
>> tucu@cloudera.com
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Neil,
>> >>>>>>
>> >>>>>> Are you trying to use the new MR API?
>> >>>>>>
>> >>>>>> If so please look at
>> >>>>>>
>> >>>>>> https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
>> >>>>>>
>> >>>>>> The "How to run Map-reduce job written using new Hadoop API?"
>> section.
>> >>>>>>
>> >>>>>> Thanks.
>> >>>>>>
>> >>>>>> Alejandro
>> >>>>>>
>> >>>>>> On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <
>> >>>> neilyalowitz@gmail.com>
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>> Hi all,
>> >>>>>>>
>> >>>>>>> I'm attempting to pass a property to an MR job with an oozie
>> workflow
>> >>>>>>> structured with a <map-reduce> tag. This was trivial with a
>> <java>
>> >>>> tag,
>> >>>>>> as
>> >>>>>>> I would call my own main(String[] args) and instantiate my own
>> >>>> Driver and
>> >>>>>>> then ToolRunner.run(). Because I was coding my own main(), I had
>> an
>> >>>>>>> opportunity to parse/set properties easily:
>> >>>>>>>
>> >>>>>>> workflow.xml
>> >>>>>>> <java>
>> >>>>>>> <main-class>com.acme.MyMain</main-class>
>> >>>>>>> <configuration>
>> >>>>>>> ...
>> >>>>>>> </configuration>
>> >>>>>>> <arg>foovalue</arg>
>> >>>>>>> <arg>barvalue</arg>
>> >>>>>>> </java>
>> >>>>>>>
>> >>>>>>> ...and capture those arguments:
>> >>>>>>>
>> >>>>>>> MyMain.java
>> >>>>>>> public static void main(String[] args) throws Exception {
>> >>>>>>> ...
>> >>>>>>> String foo = args[0];
>> >>>>>>> String bar = args[1];
>> >>>>>>> String[] argsAll = {foo, bar};
>> >>>>>>> Configuration conf = new Configuration();
>> >>>>>>> ToolRunner.run(conf, new MyDriver(), argsAll);
>> >>>>>>> }
>> >>>>>>>
>> >>>>>>> This worked fine, but I can't figure out how to capture those
>> >>>> properties
>> >>>>>>> using a <map-reduce> tag. I attempt something like this:
>> >>>>>>>
>> >>>>>>> workflow.xml
>> >>>>>>> <map-reduce>
>> >>>>>>> ...
>> >>>>>>> <configuration>
>> >>>>>>> <property>
>> >>>>>>> <name>foo</name>
>> >>>>>>> <value>foovalue</value>
>> >>>>>>> </property>
>> >>>>>>> </configuration>
>> >>>>>>> </map-reduce>
>> >>>>>>>
>> >>>>>>> However, the foo property is not set in the created
>> >>>>>>> org.apache.hadoop.conf.Configuration object:
>> >>>>>>>
>> >>>>>>> MyMapper.java
>> >>>>>>> ...
>> >>>>>>> @Override
>> >>>>>>> protected void setup(Context context)... {
>> >>>>>>> context.getConfiguration().get("foo"); //null!
>> >>>>>>> }
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Is there a way to pass an arbitrary property (ie-- not a mapred.*
>> >>>>>> property)
>> >>>>>>> to a job with an Oozie workflow using a <map-reduce> tag?
>> Perhaps I
>> >>>> am
>> >>>>>>> missing something obvious...
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Thanks!
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Neil Yalowitz
>> >>>>>>
>> >>>>
>> >>>
>> >>>
>> >
>>
>>
>
Re: passing an arbitrary property to a job with map-reduce
Posted by Neil Yalowitz <ne...@gmail.com>.
Ok, good to know. Alejandro, Mohammad, Virag thanks for the help.
Neil Yalowitz
On Wed, Jan 25, 2012 at 2:18 PM, Virag Kothari <vi...@yahoo-inc.com> wrote:
> Hey Neil,
>
> If you using the old API, you have to override the configure() of Mapper.
>
> E.g
>
> import org.apache.hadoop.mapred.Mapper;
>
> public class SampleMapper implements Mapper<LongWritable, Text,
> LongWritable, Text> {
>
> public void configure(JobConf jobConf) {
> System.out.println(jobConf.get("foo")); //should work
> }
> }
>
> Thanks,
> Virag
>
>
> On 1/25/12 11:09 AM, "Mohammad Islam" <mi...@yahoo.com> wrote:
>
> > One of our dev(API) has successfully tested this case with old API.
> > I would ask Virag to share his findings/usage preferably as code snippet.
> >
> > Regards,
> > Mohammad
> >
> >
> > ----- Original Message -----
> > From: Alejandro Abdelnur <tu...@cloudera.com>
> > To: oozie-users@incubator.apache.org
> > Cc:
> > Sent: Wednesday, January 25, 2012 10:45 AM
> > Subject: Re: passing an arbitrary property to a job with map-reduce
> >
> > Neil,
> >
> > It looks you are using the new MR API for your mapper class.
> >
> > I'd suggest start from the Oozie MR example working and modified it to
> > verify that property is propagated.
> >
> > Thanks.
> >
> > Alejadnro
> >
> > On Wed, Jan 25, 2012 at 10:12 AM, Neil Yalowitz <ne...@gmail.com>
> > wrote:
> >> Forgot to mention: I'm testing with Oozie v3.0.0 (but our production
> system
> >> is actually v3.0.2).
> >>
> >> Thanks,
> >> Neil
> >>
> >> On Wed, Jan 25, 2012 at 12:55 PM, Neil Yalowitz
> >> <ne...@gmail.com>wrote:
> >>
> >>> Yes, see these Pastebin links:
> >>>
> >>>
> >>> workflow.xml
> >>> http://pastebin.com/MqXwecjQ
> >>>
> >>> ACMEMapper.java
> >>> http://pastebin.com/Y9rFcU97
> >>>
> >>>
> >>> Thanks!
> >>> Neil
> >>>
> >>> On Wed, Jan 25, 2012 at 12:23 PM, Alejandro Abdelnur
> >>> <tu...@cloudera.com>wrote:
> >>>
> >>>> Neil,
> >>>>
> >>>> You may be missing something. This should work.
> >>>>
> >>>> would you mind sharing a minimal WF.xml and MR java that shows your
> >>>> problem?
> >>>>
> >>>> Thanks.
> >>>>
> >>>> Alejandro
> >>>>
> >>>> On Wed, Jan 25, 2012 at 9:03 AM, Neil Yalowitz <
> neilyalowitz@gmail.com>
> >>>> wrote:
> >>>>> Alejandro,
> >>>>>
> >>>>> Yes, we were using the new MR API, so I modified the workflow.xml to
> >>>> remove
> >>>>> mapred.mapper.new-api and re-tested. I still cannot pass a property
> to
> >>>> my
> >>>>> MR job.
> >>>>>
> >>>>> Is passing a non mapred.* property to a <map-reduce> job supported
> with
> >>>>> Oozie? Or must I use the <java> job and parse the property in my
> >>>> main()?
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>> Neil
> >>>>>
> >>>>> On Tue, Jan 24, 2012 at 6:00 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> >>>>> wrote:
> >>>>>
> >>>>>> Neil,
> >>>>>>
> >>>>>> Are you trying to use the new MR API?
> >>>>>>
> >>>>>> If so please look at
> >>>>>>
> >>>>>> https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
> >>>>>>
> >>>>>> The "How to run Map-reduce job written using new Hadoop API?"
> section.
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>> Alejandro
> >>>>>>
> >>>>>> On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <
> >>>> neilyalowitz@gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> I'm attempting to pass a property to an MR job with an oozie
> workflow
> >>>>>>> structured with a <map-reduce> tag. This was trivial with a <java>
> >>>> tag,
> >>>>>> as
> >>>>>>> I would call my own main(String[] args) and instantiate my own
> >>>> Driver and
> >>>>>>> then ToolRunner.run(). Because I was coding my own main(), I had
> an
> >>>>>>> opportunity to parse/set properties easily:
> >>>>>>>
> >>>>>>> workflow.xml
> >>>>>>> <java>
> >>>>>>> <main-class>com.acme.MyMain</main-class>
> >>>>>>> <configuration>
> >>>>>>> ...
> >>>>>>> </configuration>
> >>>>>>> <arg>foovalue</arg>
> >>>>>>> <arg>barvalue</arg>
> >>>>>>> </java>
> >>>>>>>
> >>>>>>> ...and capture those arguments:
> >>>>>>>
> >>>>>>> MyMain.java
> >>>>>>> public static void main(String[] args) throws Exception {
> >>>>>>> ...
> >>>>>>> String foo = args[0];
> >>>>>>> String bar = args[1];
> >>>>>>> String[] argsAll = {foo, bar};
> >>>>>>> Configuration conf = new Configuration();
> >>>>>>> ToolRunner.run(conf, new MyDriver(), argsAll);
> >>>>>>> }
> >>>>>>>
> >>>>>>> This worked fine, but I can't figure out how to capture those
> >>>> properties
> >>>>>>> using a <map-reduce> tag. I attempt something like this:
> >>>>>>>
> >>>>>>> workflow.xml
> >>>>>>> <map-reduce>
> >>>>>>> ...
> >>>>>>> <configuration>
> >>>>>>> <property>
> >>>>>>> <name>foo</name>
> >>>>>>> <value>foovalue</value>
> >>>>>>> </property>
> >>>>>>> </configuration>
> >>>>>>> </map-reduce>
> >>>>>>>
> >>>>>>> However, the foo property is not set in the created
> >>>>>>> org.apache.hadoop.conf.Configuration object:
> >>>>>>>
> >>>>>>> MyMapper.java
> >>>>>>> ...
> >>>>>>> @Override
> >>>>>>> protected void setup(Context context)... {
> >>>>>>> context.getConfiguration().get("foo"); //null!
> >>>>>>> }
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Is there a way to pass an arbitrary property (ie-- not a mapred.*
> >>>>>> property)
> >>>>>>> to a job with an Oozie workflow using a <map-reduce> tag? Perhaps
> I
> >>>> am
> >>>>>>> missing something obvious...
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>>
> >>>>>>> Neil Yalowitz
> >>>>>>
> >>>>
> >>>
> >>>
> >
>
>
Re: passing an arbitrary property to a job with map-reduce
Posted by Virag Kothari <vi...@yahoo-inc.com>.
Hey Neil,
If you using the old API, you have to override the configure() of Mapper.
E.g
import org.apache.hadoop.mapred.Mapper;
public class SampleMapper implements Mapper<LongWritable, Text,
LongWritable, Text> {
public void configure(JobConf jobConf) {
System.out.println(jobConf.get("foo")); //should work
}
}
Thanks,
Virag
On 1/25/12 11:09 AM, "Mohammad Islam" <mi...@yahoo.com> wrote:
> One of our dev(API) has successfully tested this case with old API.
> I would ask Virag to share his findings/usage preferably as code snippet.
>
> Regards,
> Mohammad
>
>
> ----- Original Message -----
> From: Alejandro Abdelnur <tu...@cloudera.com>
> To: oozie-users@incubator.apache.org
> Cc:
> Sent: Wednesday, January 25, 2012 10:45 AM
> Subject: Re: passing an arbitrary property to a job with map-reduce
>
> Neil,
>
> It looks you are using the new MR API for your mapper class.
>
> I'd suggest start from the Oozie MR example working and modified it to
> verify that property is propagated.
>
> Thanks.
>
> Alejadnro
>
> On Wed, Jan 25, 2012 at 10:12 AM, Neil Yalowitz <ne...@gmail.com>
> wrote:
>> Forgot to mention: I'm testing with Oozie v3.0.0 (but our production system
>> is actually v3.0.2).
>>
>> Thanks,
>> Neil
>>
>> On Wed, Jan 25, 2012 at 12:55 PM, Neil Yalowitz
>> <ne...@gmail.com>wrote:
>>
>>> Yes, see these Pastebin links:
>>>
>>>
>>> workflow.xml
>>> http://pastebin.com/MqXwecjQ
>>>
>>> ACMEMapper.java
>>> http://pastebin.com/Y9rFcU97
>>>
>>>
>>> Thanks!
>>> Neil
>>>
>>> On Wed, Jan 25, 2012 at 12:23 PM, Alejandro Abdelnur
>>> <tu...@cloudera.com>wrote:
>>>
>>>> Neil,
>>>>
>>>> You may be missing something. This should work.
>>>>
>>>> would you mind sharing a minimal WF.xml and MR java that shows your
>>>> problem?
>>>>
>>>> Thanks.
>>>>
>>>> Alejandro
>>>>
>>>> On Wed, Jan 25, 2012 at 9:03 AM, Neil Yalowitz <ne...@gmail.com>
>>>> wrote:
>>>>> Alejandro,
>>>>>
>>>>> Yes, we were using the new MR API, so I modified the workflow.xml to
>>>> remove
>>>>> mapred.mapper.new-api and re-tested. I still cannot pass a property to
>>>> my
>>>>> MR job.
>>>>>
>>>>> Is passing a non mapred.* property to a <map-reduce> job supported with
>>>>> Oozie? Or must I use the <java> job and parse the property in my
>>>> main()?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Neil
>>>>>
>>>>> On Tue, Jan 24, 2012 at 6:00 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>>> wrote:
>>>>>
>>>>>> Neil,
>>>>>>
>>>>>> Are you trying to use the new MR API?
>>>>>>
>>>>>> If so please look at
>>>>>>
>>>>>> https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
>>>>>>
>>>>>> The "How to run Map-reduce job written using new Hadoop API?" section.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> Alejandro
>>>>>>
>>>>>> On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <
>>>> neilyalowitz@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I'm attempting to pass a property to an MR job with an oozie workflow
>>>>>>> structured with a <map-reduce> tag. This was trivial with a <java>
>>>> tag,
>>>>>> as
>>>>>>> I would call my own main(String[] args) and instantiate my own
>>>> Driver and
>>>>>>> then ToolRunner.run(). Because I was coding my own main(), I had an
>>>>>>> opportunity to parse/set properties easily:
>>>>>>>
>>>>>>> workflow.xml
>>>>>>> <java>
>>>>>>> <main-class>com.acme.MyMain</main-class>
>>>>>>> <configuration>
>>>>>>> ...
>>>>>>> </configuration>
>>>>>>> <arg>foovalue</arg>
>>>>>>> <arg>barvalue</arg>
>>>>>>> </java>
>>>>>>>
>>>>>>> ...and capture those arguments:
>>>>>>>
>>>>>>> MyMain.java
>>>>>>> public static void main(String[] args) throws Exception {
>>>>>>> ...
>>>>>>> String foo = args[0];
>>>>>>> String bar = args[1];
>>>>>>> String[] argsAll = {foo, bar};
>>>>>>> Configuration conf = new Configuration();
>>>>>>> ToolRunner.run(conf, new MyDriver(), argsAll);
>>>>>>> }
>>>>>>>
>>>>>>> This worked fine, but I can't figure out how to capture those
>>>> properties
>>>>>>> using a <map-reduce> tag. I attempt something like this:
>>>>>>>
>>>>>>> workflow.xml
>>>>>>> <map-reduce>
>>>>>>> ...
>>>>>>> <configuration>
>>>>>>> <property>
>>>>>>> <name>foo</name>
>>>>>>> <value>foovalue</value>
>>>>>>> </property>
>>>>>>> </configuration>
>>>>>>> </map-reduce>
>>>>>>>
>>>>>>> However, the foo property is not set in the created
>>>>>>> org.apache.hadoop.conf.Configuration object:
>>>>>>>
>>>>>>> MyMapper.java
>>>>>>> ...
>>>>>>> @Override
>>>>>>> protected void setup(Context context)... {
>>>>>>> context.getConfiguration().get("foo"); //null!
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Is there a way to pass an arbitrary property (ie-- not a mapred.*
>>>>>> property)
>>>>>>> to a job with an Oozie workflow using a <map-reduce> tag? Perhaps I
>>>> am
>>>>>>> missing something obvious...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>> Neil Yalowitz
>>>>>>
>>>>
>>>
>>>
>
Re: passing an arbitrary property to a job with map-reduce
Posted by Mohammad Islam <mi...@yahoo.com>.
One of our dev(API) has successfully tested this case with old API.
I would ask Virag to share his findings/usage preferably as code snippet.
Regards,
Mohammad
----- Original Message -----
From: Alejandro Abdelnur <tu...@cloudera.com>
To: oozie-users@incubator.apache.org
Cc:
Sent: Wednesday, January 25, 2012 10:45 AM
Subject: Re: passing an arbitrary property to a job with map-reduce
Neil,
It looks you are using the new MR API for your mapper class.
I'd suggest start from the Oozie MR example working and modified it to
verify that property is propagated.
Thanks.
Alejadnro
On Wed, Jan 25, 2012 at 10:12 AM, Neil Yalowitz <ne...@gmail.com> wrote:
> Forgot to mention: I'm testing with Oozie v3.0.0 (but our production system
> is actually v3.0.2).
>
> Thanks,
> Neil
>
> On Wed, Jan 25, 2012 at 12:55 PM, Neil Yalowitz <ne...@gmail.com>wrote:
>
>> Yes, see these Pastebin links:
>>
>>
>> workflow.xml
>> http://pastebin.com/MqXwecjQ
>>
>> ACMEMapper.java
>> http://pastebin.com/Y9rFcU97
>>
>>
>> Thanks!
>> Neil
>>
>> On Wed, Jan 25, 2012 at 12:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> Neil,
>>>
>>> You may be missing something. This should work.
>>>
>>> would you mind sharing a minimal WF.xml and MR java that shows your
>>> problem?
>>>
>>> Thanks.
>>>
>>> Alejandro
>>>
>>> On Wed, Jan 25, 2012 at 9:03 AM, Neil Yalowitz <ne...@gmail.com>
>>> wrote:
>>> > Alejandro,
>>> >
>>> > Yes, we were using the new MR API, so I modified the workflow.xml to
>>> remove
>>> > mapred.mapper.new-api and re-tested. I still cannot pass a property to
>>> my
>>> > MR job.
>>> >
>>> > Is passing a non mapred.* property to a <map-reduce> job supported with
>>> > Oozie? Or must I use the <java> job and parse the property in my
>>> main()?
>>> >
>>> >
>>> > Thanks,
>>> > Neil
>>> >
>>> > On Tue, Jan 24, 2012 at 6:00 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> >wrote:
>>> >
>>> >> Neil,
>>> >>
>>> >> Are you trying to use the new MR API?
>>> >>
>>> >> If so please look at
>>> >>
>>> >> https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
>>> >>
>>> >> The "How to run Map-reduce job written using new Hadoop API?" section.
>>> >>
>>> >> Thanks.
>>> >>
>>> >> Alejandro
>>> >>
>>> >> On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <
>>> neilyalowitz@gmail.com>
>>> >> wrote:
>>> >> >
>>> >> > Hi all,
>>> >> >
>>> >> > I'm attempting to pass a property to an MR job with an oozie workflow
>>> >> > structured with a <map-reduce> tag. This was trivial with a <java>
>>> tag,
>>> >> as
>>> >> > I would call my own main(String[] args) and instantiate my own
>>> Driver and
>>> >> > then ToolRunner.run(). Because I was coding my own main(), I had an
>>> >> > opportunity to parse/set properties easily:
>>> >> >
>>> >> > workflow.xml
>>> >> > <java>
>>> >> > <main-class>com.acme.MyMain</main-class>
>>> >> > <configuration>
>>> >> > ...
>>> >> > </configuration>
>>> >> > <arg>foovalue</arg>
>>> >> > <arg>barvalue</arg>
>>> >> > </java>
>>> >> >
>>> >> > ...and capture those arguments:
>>> >> >
>>> >> > MyMain.java
>>> >> > public static void main(String[] args) throws Exception {
>>> >> > ...
>>> >> > String foo = args[0];
>>> >> > String bar = args[1];
>>> >> > String[] argsAll = {foo, bar};
>>> >> > Configuration conf = new Configuration();
>>> >> > ToolRunner.run(conf, new MyDriver(), argsAll);
>>> >> > }
>>> >> >
>>> >> > This worked fine, but I can't figure out how to capture those
>>> properties
>>> >> > using a <map-reduce> tag. I attempt something like this:
>>> >> >
>>> >> > workflow.xml
>>> >> > <map-reduce>
>>> >> > ...
>>> >> > <configuration>
>>> >> > <property>
>>> >> > <name>foo</name>
>>> >> > <value>foovalue</value>
>>> >> > </property>
>>> >> > </configuration>
>>> >> > </map-reduce>
>>> >> >
>>> >> > However, the foo property is not set in the created
>>> >> > org.apache.hadoop.conf.Configuration object:
>>> >> >
>>> >> > MyMapper.java
>>> >> > ...
>>> >> > @Override
>>> >> > protected void setup(Context context)... {
>>> >> > context.getConfiguration().get("foo"); //null!
>>> >> > }
>>> >> >
>>> >> >
>>> >> >
>>> >> > Is there a way to pass an arbitrary property (ie-- not a mapred.*
>>> >> property)
>>> >> > to a job with an Oozie workflow using a <map-reduce> tag? Perhaps I
>>> am
>>> >> > missing something obvious...
>>> >> >
>>> >> >
>>> >> >
>>> >> > Thanks!
>>> >> >
>>> >> >
>>> >> > Neil Yalowitz
>>> >>
>>>
>>
>>
Re: passing an arbitrary property to a job with map-reduce
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Neil,
It looks you are using the new MR API for your mapper class.
I'd suggest start from the Oozie MR example working and modified it to
verify that property is propagated.
Thanks.
Alejadnro
On Wed, Jan 25, 2012 at 10:12 AM, Neil Yalowitz <ne...@gmail.com> wrote:
> Forgot to mention: I'm testing with Oozie v3.0.0 (but our production system
> is actually v3.0.2).
>
> Thanks,
> Neil
>
> On Wed, Jan 25, 2012 at 12:55 PM, Neil Yalowitz <ne...@gmail.com>wrote:
>
>> Yes, see these Pastebin links:
>>
>>
>> workflow.xml
>> http://pastebin.com/MqXwecjQ
>>
>> ACMEMapper.java
>> http://pastebin.com/Y9rFcU97
>>
>>
>> Thanks!
>> Neil
>>
>> On Wed, Jan 25, 2012 at 12:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> Neil,
>>>
>>> You may be missing something. This should work.
>>>
>>> would you mind sharing a minimal WF.xml and MR java that shows your
>>> problem?
>>>
>>> Thanks.
>>>
>>> Alejandro
>>>
>>> On Wed, Jan 25, 2012 at 9:03 AM, Neil Yalowitz <ne...@gmail.com>
>>> wrote:
>>> > Alejandro,
>>> >
>>> > Yes, we were using the new MR API, so I modified the workflow.xml to
>>> remove
>>> > mapred.mapper.new-api and re-tested. I still cannot pass a property to
>>> my
>>> > MR job.
>>> >
>>> > Is passing a non mapred.* property to a <map-reduce> job supported with
>>> > Oozie? Or must I use the <java> job and parse the property in my
>>> main()?
>>> >
>>> >
>>> > Thanks,
>>> > Neil
>>> >
>>> > On Tue, Jan 24, 2012 at 6:00 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> >wrote:
>>> >
>>> >> Neil,
>>> >>
>>> >> Are you trying to use the new MR API?
>>> >>
>>> >> If so please look at
>>> >>
>>> >> https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
>>> >>
>>> >> The "How to run Map-reduce job written using new Hadoop API?" section.
>>> >>
>>> >> Thanks.
>>> >>
>>> >> Alejandro
>>> >>
>>> >> On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <
>>> neilyalowitz@gmail.com>
>>> >> wrote:
>>> >> >
>>> >> > Hi all,
>>> >> >
>>> >> > I'm attempting to pass a property to an MR job with an oozie workflow
>>> >> > structured with a <map-reduce> tag. This was trivial with a <java>
>>> tag,
>>> >> as
>>> >> > I would call my own main(String[] args) and instantiate my own
>>> Driver and
>>> >> > then ToolRunner.run(). Because I was coding my own main(), I had an
>>> >> > opportunity to parse/set properties easily:
>>> >> >
>>> >> > workflow.xml
>>> >> > <java>
>>> >> > <main-class>com.acme.MyMain</main-class>
>>> >> > <configuration>
>>> >> > ...
>>> >> > </configuration>
>>> >> > <arg>foovalue</arg>
>>> >> > <arg>barvalue</arg>
>>> >> > </java>
>>> >> >
>>> >> > ...and capture those arguments:
>>> >> >
>>> >> > MyMain.java
>>> >> > public static void main(String[] args) throws Exception {
>>> >> > ...
>>> >> > String foo = args[0];
>>> >> > String bar = args[1];
>>> >> > String[] argsAll = {foo, bar};
>>> >> > Configuration conf = new Configuration();
>>> >> > ToolRunner.run(conf, new MyDriver(), argsAll);
>>> >> > }
>>> >> >
>>> >> > This worked fine, but I can't figure out how to capture those
>>> properties
>>> >> > using a <map-reduce> tag. I attempt something like this:
>>> >> >
>>> >> > workflow.xml
>>> >> > <map-reduce>
>>> >> > ...
>>> >> > <configuration>
>>> >> > <property>
>>> >> > <name>foo</name>
>>> >> > <value>foovalue</value>
>>> >> > </property>
>>> >> > </configuration>
>>> >> > </map-reduce>
>>> >> >
>>> >> > However, the foo property is not set in the created
>>> >> > org.apache.hadoop.conf.Configuration object:
>>> >> >
>>> >> > MyMapper.java
>>> >> > ...
>>> >> > @Override
>>> >> > protected void setup(Context context)... {
>>> >> > context.getConfiguration().get("foo"); //null!
>>> >> > }
>>> >> >
>>> >> >
>>> >> >
>>> >> > Is there a way to pass an arbitrary property (ie-- not a mapred.*
>>> >> property)
>>> >> > to a job with an Oozie workflow using a <map-reduce> tag? Perhaps I
>>> am
>>> >> > missing something obvious...
>>> >> >
>>> >> >
>>> >> >
>>> >> > Thanks!
>>> >> >
>>> >> >
>>> >> > Neil Yalowitz
>>> >>
>>>
>>
>>
Re: passing an arbitrary property to a job with map-reduce
Posted by Neil Yalowitz <ne...@gmail.com>.
Forgot to mention: I'm testing with Oozie v3.0.0 (but our production system
is actually v3.0.2).
Thanks,
Neil
On Wed, Jan 25, 2012 at 12:55 PM, Neil Yalowitz <ne...@gmail.com>wrote:
> Yes, see these Pastebin links:
>
>
> workflow.xml
> http://pastebin.com/MqXwecjQ
>
> ACMEMapper.java
> http://pastebin.com/Y9rFcU97
>
>
> Thanks!
> Neil
>
> On Wed, Jan 25, 2012 at 12:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Neil,
>>
>> You may be missing something. This should work.
>>
>> would you mind sharing a minimal WF.xml and MR java that shows your
>> problem?
>>
>> Thanks.
>>
>> Alejandro
>>
>> On Wed, Jan 25, 2012 at 9:03 AM, Neil Yalowitz <ne...@gmail.com>
>> wrote:
>> > Alejandro,
>> >
>> > Yes, we were using the new MR API, so I modified the workflow.xml to
>> remove
>> > mapred.mapper.new-api and re-tested. I still cannot pass a property to
>> my
>> > MR job.
>> >
>> > Is passing a non mapred.* property to a <map-reduce> job supported with
>> > Oozie? Or must I use the <java> job and parse the property in my
>> main()?
>> >
>> >
>> > Thanks,
>> > Neil
>> >
>> > On Tue, Jan 24, 2012 at 6:00 PM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>> >
>> >> Neil,
>> >>
>> >> Are you trying to use the new MR API?
>> >>
>> >> If so please look at
>> >>
>> >> https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
>> >>
>> >> The "How to run Map-reduce job written using new Hadoop API?" section.
>> >>
>> >> Thanks.
>> >>
>> >> Alejandro
>> >>
>> >> On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <
>> neilyalowitz@gmail.com>
>> >> wrote:
>> >> >
>> >> > Hi all,
>> >> >
>> >> > I'm attempting to pass a property to an MR job with an oozie workflow
>> >> > structured with a <map-reduce> tag. This was trivial with a <java>
>> tag,
>> >> as
>> >> > I would call my own main(String[] args) and instantiate my own
>> Driver and
>> >> > then ToolRunner.run(). Because I was coding my own main(), I had an
>> >> > opportunity to parse/set properties easily:
>> >> >
>> >> > workflow.xml
>> >> > <java>
>> >> > <main-class>com.acme.MyMain</main-class>
>> >> > <configuration>
>> >> > ...
>> >> > </configuration>
>> >> > <arg>foovalue</arg>
>> >> > <arg>barvalue</arg>
>> >> > </java>
>> >> >
>> >> > ...and capture those arguments:
>> >> >
>> >> > MyMain.java
>> >> > public static void main(String[] args) throws Exception {
>> >> > ...
>> >> > String foo = args[0];
>> >> > String bar = args[1];
>> >> > String[] argsAll = {foo, bar};
>> >> > Configuration conf = new Configuration();
>> >> > ToolRunner.run(conf, new MyDriver(), argsAll);
>> >> > }
>> >> >
>> >> > This worked fine, but I can't figure out how to capture those
>> properties
>> >> > using a <map-reduce> tag. I attempt something like this:
>> >> >
>> >> > workflow.xml
>> >> > <map-reduce>
>> >> > ...
>> >> > <configuration>
>> >> > <property>
>> >> > <name>foo</name>
>> >> > <value>foovalue</value>
>> >> > </property>
>> >> > </configuration>
>> >> > </map-reduce>
>> >> >
>> >> > However, the foo property is not set in the created
>> >> > org.apache.hadoop.conf.Configuration object:
>> >> >
>> >> > MyMapper.java
>> >> > ...
>> >> > @Override
>> >> > protected void setup(Context context)... {
>> >> > context.getConfiguration().get("foo"); //null!
>> >> > }
>> >> >
>> >> >
>> >> >
>> >> > Is there a way to pass an arbitrary property (ie-- not a mapred.*
>> >> property)
>> >> > to a job with an Oozie workflow using a <map-reduce> tag? Perhaps I
>> am
>> >> > missing something obvious...
>> >> >
>> >> >
>> >> >
>> >> > Thanks!
>> >> >
>> >> >
>> >> > Neil Yalowitz
>> >>
>>
>
>
Re: passing an arbitrary property to a job with map-reduce
Posted by Neil Yalowitz <ne...@gmail.com>.
Yes, see these Pastebin links:
workflow.xml
http://pastebin.com/MqXwecjQ
ACMEMapper.java
http://pastebin.com/Y9rFcU97
Thanks!
Neil
On Wed, Jan 25, 2012 at 12:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Neil,
>
> You may be missing something. This should work.
>
> would you mind sharing a minimal WF.xml and MR java that shows your
> problem?
>
> Thanks.
>
> Alejandro
>
> On Wed, Jan 25, 2012 at 9:03 AM, Neil Yalowitz <ne...@gmail.com>
> wrote:
> > Alejandro,
> >
> > Yes, we were using the new MR API, so I modified the workflow.xml to
> remove
> > mapred.mapper.new-api and re-tested. I still cannot pass a property to
> my
> > MR job.
> >
> > Is passing a non mapred.* property to a <map-reduce> job supported with
> > Oozie? Or must I use the <java> job and parse the property in my main()?
> >
> >
> > Thanks,
> > Neil
> >
> > On Tue, Jan 24, 2012 at 6:00 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >
> >> Neil,
> >>
> >> Are you trying to use the new MR API?
> >>
> >> If so please look at
> >>
> >> https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
> >>
> >> The "How to run Map-reduce job written using new Hadoop API?" section.
> >>
> >> Thanks.
> >>
> >> Alejandro
> >>
> >> On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <neilyalowitz@gmail.com
> >
> >> wrote:
> >> >
> >> > Hi all,
> >> >
> >> > I'm attempting to pass a property to an MR job with an oozie workflow
> >> > structured with a <map-reduce> tag. This was trivial with a <java>
> tag,
> >> as
> >> > I would call my own main(String[] args) and instantiate my own Driver
> and
> >> > then ToolRunner.run(). Because I was coding my own main(), I had an
> >> > opportunity to parse/set properties easily:
> >> >
> >> > workflow.xml
> >> > <java>
> >> > <main-class>com.acme.MyMain</main-class>
> >> > <configuration>
> >> > ...
> >> > </configuration>
> >> > <arg>foovalue</arg>
> >> > <arg>barvalue</arg>
> >> > </java>
> >> >
> >> > ...and capture those arguments:
> >> >
> >> > MyMain.java
> >> > public static void main(String[] args) throws Exception {
> >> > ...
> >> > String foo = args[0];
> >> > String bar = args[1];
> >> > String[] argsAll = {foo, bar};
> >> > Configuration conf = new Configuration();
> >> > ToolRunner.run(conf, new MyDriver(), argsAll);
> >> > }
> >> >
> >> > This worked fine, but I can't figure out how to capture those
> properties
> >> > using a <map-reduce> tag. I attempt something like this:
> >> >
> >> > workflow.xml
> >> > <map-reduce>
> >> > ...
> >> > <configuration>
> >> > <property>
> >> > <name>foo</name>
> >> > <value>foovalue</value>
> >> > </property>
> >> > </configuration>
> >> > </map-reduce>
> >> >
> >> > However, the foo property is not set in the created
> >> > org.apache.hadoop.conf.Configuration object:
> >> >
> >> > MyMapper.java
> >> > ...
> >> > @Override
> >> > protected void setup(Context context)... {
> >> > context.getConfiguration().get("foo"); //null!
> >> > }
> >> >
> >> >
> >> >
> >> > Is there a way to pass an arbitrary property (ie-- not a mapred.*
> >> property)
> >> > to a job with an Oozie workflow using a <map-reduce> tag? Perhaps I
> am
> >> > missing something obvious...
> >> >
> >> >
> >> >
> >> > Thanks!
> >> >
> >> >
> >> > Neil Yalowitz
> >>
>
Re: passing an arbitrary property to a job with map-reduce
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Neil,
You may be missing something. This should work.
would you mind sharing a minimal WF.xml and MR java that shows your problem?
Thanks.
Alejandro
On Wed, Jan 25, 2012 at 9:03 AM, Neil Yalowitz <ne...@gmail.com> wrote:
> Alejandro,
>
> Yes, we were using the new MR API, so I modified the workflow.xml to remove
> mapred.mapper.new-api and re-tested. I still cannot pass a property to my
> MR job.
>
> Is passing a non mapred.* property to a <map-reduce> job supported with
> Oozie? Or must I use the <java> job and parse the property in my main()?
>
>
> Thanks,
> Neil
>
> On Tue, Jan 24, 2012 at 6:00 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Neil,
>>
>> Are you trying to use the new MR API?
>>
>> If so please look at
>>
>> https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
>>
>> The "How to run Map-reduce job written using new Hadoop API?" section.
>>
>> Thanks.
>>
>> Alejandro
>>
>> On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <ne...@gmail.com>
>> wrote:
>> >
>> > Hi all,
>> >
>> > I'm attempting to pass a property to an MR job with an oozie workflow
>> > structured with a <map-reduce> tag. This was trivial with a <java> tag,
>> as
>> > I would call my own main(String[] args) and instantiate my own Driver and
>> > then ToolRunner.run(). Because I was coding my own main(), I had an
>> > opportunity to parse/set properties easily:
>> >
>> > workflow.xml
>> > <java>
>> > <main-class>com.acme.MyMain</main-class>
>> > <configuration>
>> > ...
>> > </configuration>
>> > <arg>foovalue</arg>
>> > <arg>barvalue</arg>
>> > </java>
>> >
>> > ...and capture those arguments:
>> >
>> > MyMain.java
>> > public static void main(String[] args) throws Exception {
>> > ...
>> > String foo = args[0];
>> > String bar = args[1];
>> > String[] argsAll = {foo, bar};
>> > Configuration conf = new Configuration();
>> > ToolRunner.run(conf, new MyDriver(), argsAll);
>> > }
>> >
>> > This worked fine, but I can't figure out how to capture those properties
>> > using a <map-reduce> tag. I attempt something like this:
>> >
>> > workflow.xml
>> > <map-reduce>
>> > ...
>> > <configuration>
>> > <property>
>> > <name>foo</name>
>> > <value>foovalue</value>
>> > </property>
>> > </configuration>
>> > </map-reduce>
>> >
>> > However, the foo property is not set in the created
>> > org.apache.hadoop.conf.Configuration object:
>> >
>> > MyMapper.java
>> > ...
>> > @Override
>> > protected void setup(Context context)... {
>> > context.getConfiguration().get("foo"); //null!
>> > }
>> >
>> >
>> >
>> > Is there a way to pass an arbitrary property (ie-- not a mapred.*
>> property)
>> > to a job with an Oozie workflow using a <map-reduce> tag? Perhaps I am
>> > missing something obvious...
>> >
>> >
>> >
>> > Thanks!
>> >
>> >
>> > Neil Yalowitz
>>
Re: passing an arbitrary property to a job with map-reduce
Posted by Neil Yalowitz <ne...@gmail.com>.
Alejandro,
Yes, we were using the new MR API, so I modified the workflow.xml to remove
mapred.mapper.new-api and re-tested. I still cannot pass a property to my
MR job.
Is passing a non mapred.* property to a <map-reduce> job supported with
Oozie? Or must I use the <java> job and parse the property in my main()?
Thanks,
Neil
On Tue, Jan 24, 2012 at 6:00 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Neil,
>
> Are you trying to use the new MR API?
>
> If so please look at
>
> https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
>
> The "How to run Map-reduce job written using new Hadoop API?" section.
>
> Thanks.
>
> Alejandro
>
> On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <ne...@gmail.com>
> wrote:
> >
> > Hi all,
> >
> > I'm attempting to pass a property to an MR job with an oozie workflow
> > structured with a <map-reduce> tag. This was trivial with a <java> tag,
> as
> > I would call my own main(String[] args) and instantiate my own Driver and
> > then ToolRunner.run(). Because I was coding my own main(), I had an
> > opportunity to parse/set properties easily:
> >
> > workflow.xml
> > <java>
> > <main-class>com.acme.MyMain</main-class>
> > <configuration>
> > ...
> > </configuration>
> > <arg>foovalue</arg>
> > <arg>barvalue</arg>
> > </java>
> >
> > ...and capture those arguments:
> >
> > MyMain.java
> > public static void main(String[] args) throws Exception {
> > ...
> > String foo = args[0];
> > String bar = args[1];
> > String[] argsAll = {foo, bar};
> > Configuration conf = new Configuration();
> > ToolRunner.run(conf, new MyDriver(), argsAll);
> > }
> >
> > This worked fine, but I can't figure out how to capture those properties
> > using a <map-reduce> tag. I attempt something like this:
> >
> > workflow.xml
> > <map-reduce>
> > ...
> > <configuration>
> > <property>
> > <name>foo</name>
> > <value>foovalue</value>
> > </property>
> > </configuration>
> > </map-reduce>
> >
> > However, the foo property is not set in the created
> > org.apache.hadoop.conf.Configuration object:
> >
> > MyMapper.java
> > ...
> > @Override
> > protected void setup(Context context)... {
> > context.getConfiguration().get("foo"); //null!
> > }
> >
> >
> >
> > Is there a way to pass an arbitrary property (ie-- not a mapred.*
> property)
> > to a job with an Oozie workflow using a <map-reduce> tag? Perhaps I am
> > missing something obvious...
> >
> >
> >
> > Thanks!
> >
> >
> > Neil Yalowitz
>
Re: passing an arbitrary property to a job with map-reduce
Posted by Mohammad Islam <mi...@yahoo.com>.
+Neil.
Looks like Neil is not getting the mail correctly.
Please include his email explicitly, until it is resolved.
________________________________
From: Alejandro Abdelnur <tu...@cloudera.com>
To: oozie-users@incubator.apache.org
Sent: Tuesday, January 24, 2012 3:00 PM
Subject: Re: passing an arbitrary property to a job with map-reduce
Neil,
Are you trying to use the new MR API?
If so please look at
https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
The "How to run Map-reduce job written using new Hadoop API?" section.
Thanks.
Alejandro
On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <ne...@gmail.com> wrote:
>
> Hi all,
>
> I'm attempting to pass a property to an MR job with an oozie workflow
> structured with a <map-reduce> tag. This was trivial with a <java> tag, as
> I would call my own main(String[] args) and instantiate my own Driver and
> then ToolRunner.run(). Because I was coding my own main(), I had an
> opportunity to parse/set properties easily:
>
> workflow.xml
> <java>
> <main-class>com.acme.MyMain</main-class>
> <configuration>
> ...
> </configuration>
> <arg>foovalue</arg>
> <arg>barvalue</arg>
> </java>
>
> ...and capture those arguments:
>
> MyMain.java
> public static void main(String[] args) throws Exception {
> ...
> String foo = args[0];
> String bar = args[1];
> String[] argsAll = {foo, bar};
> Configuration conf = new Configuration();
> ToolRunner.run(conf, new MyDriver(), argsAll);
> }
>
> This worked fine, but I can't figure out how to capture those properties
> using a <map-reduce> tag. I attempt something like this:
>
> workflow.xml
> <map-reduce>
> ...
> <configuration>
> <property>
> <name>foo</name>
> <value>foovalue</value>
> </property>
> </configuration>
> </map-reduce>
>
> However, the foo property is not set in the created
> org.apache.hadoop.conf.Configuration object:
>
> MyMapper.java
> ...
> @Override
> protected void setup(Context context)... {
> context.getConfiguration().get("foo"); //null!
> }
>
>
>
> Is there a way to pass an arbitrary property (ie-- not a mapred.* property)
> to a job with an Oozie workflow using a <map-reduce> tag? Perhaps I am
> missing something obvious...
>
>
>
> Thanks!
>
>
> Neil Yalowitz
Re: passing an arbitrary property to a job with map-reduce
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Neil,
Are you trying to use the new MR API?
If so please look at
https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
The "How to run Map-reduce job written using new Hadoop API?" section.
Thanks.
Alejandro
On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <ne...@gmail.com> wrote:
>
> Hi all,
>
> I'm attempting to pass a property to an MR job with an oozie workflow
> structured with a <map-reduce> tag. This was trivial with a <java> tag, as
> I would call my own main(String[] args) and instantiate my own Driver and
> then ToolRunner.run(). Because I was coding my own main(), I had an
> opportunity to parse/set properties easily:
>
> workflow.xml
> <java>
> <main-class>com.acme.MyMain</main-class>
> <configuration>
> ...
> </configuration>
> <arg>foovalue</arg>
> <arg>barvalue</arg>
> </java>
>
> ...and capture those arguments:
>
> MyMain.java
> public static void main(String[] args) throws Exception {
> ...
> String foo = args[0];
> String bar = args[1];
> String[] argsAll = {foo, bar};
> Configuration conf = new Configuration();
> ToolRunner.run(conf, new MyDriver(), argsAll);
> }
>
> This worked fine, but I can't figure out how to capture those properties
> using a <map-reduce> tag. I attempt something like this:
>
> workflow.xml
> <map-reduce>
> ...
> <configuration>
> <property>
> <name>foo</name>
> <value>foovalue</value>
> </property>
> </configuration>
> </map-reduce>
>
> However, the foo property is not set in the created
> org.apache.hadoop.conf.Configuration object:
>
> MyMapper.java
> ...
> @Override
> protected void setup(Context context)... {
> context.getConfiguration().get("foo"); //null!
> }
>
>
>
> Is there a way to pass an arbitrary property (ie-- not a mapred.* property)
> to a job with an Oozie workflow using a <map-reduce> tag? Perhaps I am
> missing something obvious...
>
>
>
> Thanks!
>
>
> Neil Yalowitz