You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hadoop.apache.org by KayVajj <va...@gmail.com> on 2013/09/23 23:39:01 UTC

mapred Configuration Vs mapReduce configuration in Ooozie

I am implementing an oozie workflow MR job . I was not sure if I have to
use the mapred properties vs the mapreduce properties.

I have seen that if mapreduce properties were to be used, it has to be set
explicitly like

JobConf.setUseNewMapper(boolean flag)


How do I do the same in Oozie?

Also another question, if I were to use the new properties where do I find
the property names like "mapreduce.job.inputformat.class" These seem to be
hidden in some private API.

Thanks your answers are appreciated.

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Robert Kanter <rk...@cloudera.com>.

Hi,

You can use the new api in your action if you set these two properties in
the action's <configuration> section:
    <property>
      <name>mapred.mapper.new-api</name>
      <value>true</value>
    </property>
    <property>
      <name>mapred.reducer.new-api</name>
      <value>true</value>
    </property>

The JobConf.setUseNewMapper(boolean flag) simply sets the same property;
same with JobConf.setUseNewReducer(boolean flag).


 - Robert


On Mon, Sep 23, 2013 at 2:39 PM, KayVajj <va...@gmail.com> wrote:

> I am implementing an oozie workflow MR job . I was not sure if I have to
> use the mapred properties vs the mapreduce properties.
>
> I have seen that if mapreduce properties were to be used, it has to be set
> explicitly like
>
> JobConf.setUseNewMapper(boolean flag)
>
>
> How do I do the same in Oozie?
>
> Also another question, if I were to use the new properties where do I find
> the property names like "mapreduce.job.inputformat.class" These seem to be
> hidden in some private API.
>
> Thanks your answers are appreciated.
>
>
>  --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cdh-user+unsubscribe@cloudera.org.
> For more options, visit
> https://groups.google.com/a/cloudera.org/groups/opt_out.
>

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Robert Kanter <rk...@cloudera.com>.

Hi,

You can use the new api in your action if you set these two properties in
the action's <configuration> section:
    <property>
      <name>mapred.mapper.new-api</name>
      <value>true</value>
    </property>
    <property>
      <name>mapred.reducer.new-api</name>
      <value>true</value>
    </property>

The JobConf.setUseNewMapper(boolean flag) simply sets the same property;
same with JobConf.setUseNewReducer(boolean flag).


 - Robert


On Mon, Sep 23, 2013 at 2:39 PM, KayVajj <va...@gmail.com> wrote:

> I am implementing an oozie workflow MR job . I was not sure if I have to
> use the mapred properties vs the mapreduce properties.
>
> I have seen that if mapreduce properties were to be used, it has to be set
> explicitly like
>
> JobConf.setUseNewMapper(boolean flag)
>
>
> How do I do the same in Oozie?
>
> Also another question, if I were to use the new properties where do I find
> the property names like "mapreduce.job.inputformat.class" These seem to be
> hidden in some private API.
>
> Thanks your answers are appreciated.
>
>
>  --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cdh-user+unsubscribe@cloudera.org.
> For more options, visit
> https://groups.google.com/a/cloudera.org/groups/opt_out.
>

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Mohammad Islam <mi...@yahoo.com>.

+user@oozie 

- cdh-user (can't send)

________________________________
 From: Mohammad Islam <mi...@yahoo.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; "cdh-user@cloudera.org" <cd...@cloudera.org> 
Sent: Monday, September 23, 2013 3:24 PM
Subject: Re: mapred Configuration Vs mapReduce configuration in Ooozie

Hi Kay,
Do you want to use the new hadoop API?
If yes, did you check this link ?
https://cwiki.apache.org/confluence/display/OOZIE/Map+Reduce+Cookbook

"CASE-2: RUNNING MAPREDUCE USING THE NEW HADOOP API"

For your second questions,
I don't know any good such link. But this could be helpful :
http://stackoverflow.com/questions/10986633/hadoop-configuration-mapred-vs-mapreduce

Regards,
Mohammad

________________________________
 From: KayVajj <va...@gmail.com>
To: cdh-user@cloudera.org; "common-user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Monday, September 23, 2013 2:39 PM
Subject: mapred Configuration Vs mapReduce configuration in Ooozie

I am implementing an oozie workflow MR job . I was not sure if I have to use the mapred properties vs the mapreduce properties. 

I have seen that if mapreduce properties were to be used, it has to be set explicitly like

JobConf.setUseNewMapper(boolean flag)

How do I do the same in Oozie? 

Also another question, if I were to use the new properties where do I find the property names like "mapreduce.job.inputformat.class" These seem to be hidden in some private API.

Thanks your answers are appreciated.

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Mohammad Islam <mi...@yahoo.com>.

+user@oozie 

- cdh-user (can't send)

________________________________
 From: Mohammad Islam <mi...@yahoo.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; "cdh-user@cloudera.org" <cd...@cloudera.org> 
Sent: Monday, September 23, 2013 3:24 PM
Subject: Re: mapred Configuration Vs mapReduce configuration in Ooozie

Hi Kay,
Do you want to use the new hadoop API?
If yes, did you check this link ?
https://cwiki.apache.org/confluence/display/OOZIE/Map+Reduce+Cookbook

"CASE-2: RUNNING MAPREDUCE USING THE NEW HADOOP API"

For your second questions,
I don't know any good such link. But this could be helpful :
http://stackoverflow.com/questions/10986633/hadoop-configuration-mapred-vs-mapreduce

Regards,
Mohammad

________________________________
 From: KayVajj <va...@gmail.com>
To: cdh-user@cloudera.org; "common-user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Monday, September 23, 2013 2:39 PM
Subject: mapred Configuration Vs mapReduce configuration in Ooozie

I am implementing an oozie workflow MR job . I was not sure if I have to use the mapred properties vs the mapreduce properties. 

I have seen that if mapreduce properties were to be used, it has to be set explicitly like

JobConf.setUseNewMapper(boolean flag)

How do I do the same in Oozie? 

Also another question, if I were to use the new properties where do I find the property names like "mapreduce.job.inputformat.class" These seem to be hidden in some private API.

Thanks your answers are appreciated.

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Mohammad Islam <mi...@yahoo.com>.

+user@oozie 

- cdh-user (can't send)

________________________________
 From: Mohammad Islam <mi...@yahoo.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; "cdh-user@cloudera.org" <cd...@cloudera.org> 
Sent: Monday, September 23, 2013 3:24 PM
Subject: Re: mapred Configuration Vs mapReduce configuration in Ooozie

Hi Kay,
Do you want to use the new hadoop API?
If yes, did you check this link ?
https://cwiki.apache.org/confluence/display/OOZIE/Map+Reduce+Cookbook

"CASE-2: RUNNING MAPREDUCE USING THE NEW HADOOP API"

For your second questions,
I don't know any good such link. But this could be helpful :
http://stackoverflow.com/questions/10986633/hadoop-configuration-mapred-vs-mapreduce

Regards,
Mohammad

________________________________
 From: KayVajj <va...@gmail.com>
To: cdh-user@cloudera.org; "common-user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Monday, September 23, 2013 2:39 PM
Subject: mapred Configuration Vs mapReduce configuration in Ooozie

I am implementing an oozie workflow MR job . I was not sure if I have to use the mapred properties vs the mapreduce properties. 

I have seen that if mapreduce properties were to be used, it has to be set explicitly like

JobConf.setUseNewMapper(boolean flag)

How do I do the same in Oozie? 

Also another question, if I were to use the new properties where do I find the property names like "mapreduce.job.inputformat.class" These seem to be hidden in some private API.

Thanks your answers are appreciated.

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Mohammad Islam <mi...@yahoo.com>.

+user@oozie 

- cdh-user (can't send)

________________________________
 From: Mohammad Islam <mi...@yahoo.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; "cdh-user@cloudera.org" <cd...@cloudera.org> 
Sent: Monday, September 23, 2013 3:24 PM
Subject: Re: mapred Configuration Vs mapReduce configuration in Ooozie

Hi Kay,
Do you want to use the new hadoop API?
If yes, did you check this link ?
https://cwiki.apache.org/confluence/display/OOZIE/Map+Reduce+Cookbook

"CASE-2: RUNNING MAPREDUCE USING THE NEW HADOOP API"

For your second questions,
I don't know any good such link. But this could be helpful :
http://stackoverflow.com/questions/10986633/hadoop-configuration-mapred-vs-mapreduce

Regards,
Mohammad

________________________________
 From: KayVajj <va...@gmail.com>
To: cdh-user@cloudera.org; "common-user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Monday, September 23, 2013 2:39 PM
Subject: mapred Configuration Vs mapReduce configuration in Ooozie

I am implementing an oozie workflow MR job . I was not sure if I have to use the mapred properties vs the mapreduce properties. 

I have seen that if mapreduce properties were to be used, it has to be set explicitly like

JobConf.setUseNewMapper(boolean flag)

How do I do the same in Oozie? 

Also another question, if I were to use the new properties where do I find the property names like "mapreduce.job.inputformat.class" These seem to be hidden in some private API.

Thanks your answers are appreciated.

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Mohammad Islam <mi...@yahoo.com>.

+user@oozie 

- cdh-user (can't send)

________________________________
 From: Mohammad Islam <mi...@yahoo.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; "cdh-user@cloudera.org" <cd...@cloudera.org> 
Sent: Monday, September 23, 2013 3:24 PM
Subject: Re: mapred Configuration Vs mapReduce configuration in Ooozie

Hi Kay,
Do you want to use the new hadoop API?
If yes, did you check this link ?
https://cwiki.apache.org/confluence/display/OOZIE/Map+Reduce+Cookbook

"CASE-2: RUNNING MAPREDUCE USING THE NEW HADOOP API"

For your second questions,
I don't know any good such link. But this could be helpful :
http://stackoverflow.com/questions/10986633/hadoop-configuration-mapred-vs-mapreduce

Regards,
Mohammad

________________________________
 From: KayVajj <va...@gmail.com>
To: cdh-user@cloudera.org; "common-user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Monday, September 23, 2013 2:39 PM
Subject: mapred Configuration Vs mapReduce configuration in Ooozie

I am implementing an oozie workflow MR job . I was not sure if I have to use the mapred properties vs the mapreduce properties. 

I have seen that if mapreduce properties were to be used, it has to be set explicitly like

JobConf.setUseNewMapper(boolean flag)

How do I do the same in Oozie? 

Also another question, if I were to use the new properties where do I find the property names like "mapreduce.job.inputformat.class" These seem to be hidden in some private API.

Thanks your answers are appreciated.

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Mohammad Islam <mi...@yahoo.com>.

Hi Kay,
Do you want to use the new hadoop API?
If yes, did you check this link ?
https://cwiki.apache.org/confluence/display/OOZIE/Map+Reduce+Cookbook

"CASE-2: RUNNING MAPREDUCE USING THE NEW HADOOP API"

For your second questions,
I don't know any good such link. But this could be helpful :
http://stackoverflow.com/questions/10986633/hadoop-configuration-mapred-vs-mapreduce

Regards,
Mohammad


________________________________
 From: KayVajj <va...@gmail.com>
To: cdh-user@cloudera.org; "common-user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Monday, September 23, 2013 2:39 PM
Subject: mapred Configuration Vs mapReduce configuration in Ooozie
 


I am implementing an oozie workflow MR job . I was not sure if I have to use the mapred properties vs the mapreduce properties. 

I have seen that if mapreduce properties were to be used, it has to be set explicitly like

JobConf.setUseNewMapper(boolean flag)

How do I do the same in Oozie? 

Also another question, if I were to use the new properties where do I find the property names like "mapreduce.job.inputformat.class" These seem to be hidden in some private API.

Thanks your answers are appreciated.

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Mohammad Islam <mi...@yahoo.com>.

Hi Kay,
Do you want to use the new hadoop API?
If yes, did you check this link ?
https://cwiki.apache.org/confluence/display/OOZIE/Map+Reduce+Cookbook

"CASE-2: RUNNING MAPREDUCE USING THE NEW HADOOP API"

For your second questions,
I don't know any good such link. But this could be helpful :
http://stackoverflow.com/questions/10986633/hadoop-configuration-mapred-vs-mapreduce

Regards,
Mohammad


________________________________
 From: KayVajj <va...@gmail.com>
To: cdh-user@cloudera.org; "common-user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Monday, September 23, 2013 2:39 PM
Subject: mapred Configuration Vs mapReduce configuration in Ooozie
 


I am implementing an oozie workflow MR job . I was not sure if I have to use the mapred properties vs the mapreduce properties. 

I have seen that if mapreduce properties were to be used, it has to be set explicitly like

JobConf.setUseNewMapper(boolean flag)

How do I do the same in Oozie? 

Also another question, if I were to use the new properties where do I find the property names like "mapreduce.job.inputformat.class" These seem to be hidden in some private API.

Thanks your answers are appreciated.

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Mohammad Islam <mi...@yahoo.com>.

Hi Kay,
Do you want to use the new hadoop API?
If yes, did you check this link ?
https://cwiki.apache.org/confluence/display/OOZIE/Map+Reduce+Cookbook

"CASE-2: RUNNING MAPREDUCE USING THE NEW HADOOP API"

For your second questions,
I don't know any good such link. But this could be helpful :
http://stackoverflow.com/questions/10986633/hadoop-configuration-mapred-vs-mapreduce

Regards,
Mohammad


________________________________
 From: KayVajj <va...@gmail.com>
To: cdh-user@cloudera.org; "common-user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Monday, September 23, 2013 2:39 PM
Subject: mapred Configuration Vs mapReduce configuration in Ooozie
 


I am implementing an oozie workflow MR job . I was not sure if I have to use the mapred properties vs the mapreduce properties. 

I have seen that if mapreduce properties were to be used, it has to be set explicitly like

JobConf.setUseNewMapper(boolean flag)

How do I do the same in Oozie? 

Also another question, if I were to use the new properties where do I find the property names like "mapreduce.job.inputformat.class" These seem to be hidden in some private API.

Thanks your answers are appreciated.

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Robert Kanter <rk...@cloudera.com>.

Hi,

You can use the new api in your action if you set these two properties in
the action's <configuration> section:
    <property>
      <name>mapred.mapper.new-api</name>
      <value>true</value>
    </property>
    <property>
      <name>mapred.reducer.new-api</name>
      <value>true</value>
    </property>

The JobConf.setUseNewMapper(boolean flag) simply sets the same property;
same with JobConf.setUseNewReducer(boolean flag).


 - Robert


On Mon, Sep 23, 2013 at 2:39 PM, KayVajj <va...@gmail.com> wrote:

> I am implementing an oozie workflow MR job . I was not sure if I have to
> use the mapred properties vs the mapreduce properties.
>
> I have seen that if mapreduce properties were to be used, it has to be set
> explicitly like
>
> JobConf.setUseNewMapper(boolean flag)
>
>
> How do I do the same in Oozie?
>
> Also another question, if I were to use the new properties where do I find
> the property names like "mapreduce.job.inputformat.class" These seem to be
> hidden in some private API.
>
> Thanks your answers are appreciated.
>
>
>  --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cdh-user+unsubscribe@cloudera.org.
> For more options, visit
> https://groups.google.com/a/cloudera.org/groups/opt_out.
>

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Robert Kanter <rk...@cloudera.com>.

Hi,

You can use the new api in your action if you set these two properties in
the action's <configuration> section:
    <property>
      <name>mapred.mapper.new-api</name>
      <value>true</value>
    </property>
    <property>
      <name>mapred.reducer.new-api</name>
      <value>true</value>
    </property>

The JobConf.setUseNewMapper(boolean flag) simply sets the same property;
same with JobConf.setUseNewReducer(boolean flag).


 - Robert


On Mon, Sep 23, 2013 at 2:39 PM, KayVajj <va...@gmail.com> wrote:

> I am implementing an oozie workflow MR job . I was not sure if I have to
> use the mapred properties vs the mapreduce properties.
>
> I have seen that if mapreduce properties were to be used, it has to be set
> explicitly like
>
> JobConf.setUseNewMapper(boolean flag)
>
>
> How do I do the same in Oozie?
>
> Also another question, if I were to use the new properties where do I find
> the property names like "mapreduce.job.inputformat.class" These seem to be
> hidden in some private API.
>
> Thanks your answers are appreciated.
>
>
>  --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cdh-user+unsubscribe@cloudera.org.
> For more options, visit
> https://groups.google.com/a/cloudera.org/groups/opt_out.
>

Re: mapred Configuration Vs mapReduce configuration in Ooozie

Posted by Mohammad Islam <mi...@yahoo.com>.

Hi Kay,
Do you want to use the new hadoop API?
If yes, did you check this link ?
https://cwiki.apache.org/confluence/display/OOZIE/Map+Reduce+Cookbook

"CASE-2: RUNNING MAPREDUCE USING THE NEW HADOOP API"

For your second questions,
I don't know any good such link. But this could be helpful :
http://stackoverflow.com/questions/10986633/hadoop-configuration-mapred-vs-mapreduce

Regards,
Mohammad


________________________________
 From: KayVajj <va...@gmail.com>
To: cdh-user@cloudera.org; "common-user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Monday, September 23, 2013 2:39 PM
Subject: mapred Configuration Vs mapReduce configuration in Ooozie
 


I am implementing an oozie workflow MR job . I was not sure if I have to use the mapred properties vs the mapreduce properties. 

I have seen that if mapreduce properties were to be used, it has to be set explicitly like

JobConf.setUseNewMapper(boolean flag)

How do I do the same in Oozie? 

Also another question, if I were to use the new properties where do I find the property names like "mapreduce.job.inputformat.class" These seem to be hidden in some private API.

Thanks your answers are appreciated.