You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Harshal Vora <ha...@komli.com> on 2011/12/01 16:07:22 UTC
hadoop - oozie versioning issues
Hey,
We have a map reduce job which is running perfectly well without oozie.
Now we are trying to integrate it with oozie.
If i set mapred.mapper/reducer.new-api to true, i am getting the following exception
java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:871)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
And if i comment that part, I am getting the following exception
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
....
....
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class net.atomex.amt.segcalc.preprocessor.pixel.mapreduce.PixelLogProcessorMapper not org.apache.hadoop.mapred.Mapper
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1028)
at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:968)
I tried to google the exceptions. I learned that for the Error in configuring object, we should add the new-api property in the oozie config.
And for the Type mismatch in key from map, somewhere I read that oozie is not compatible with new-api, which i do not think is the case.
I have set mapred.mapoutput/output.key.class, to Text and the value class to NullWritable in a config-default.xml as below, and
config-default is read using <job-xml>.
<property>
<name>mapred.mapoutput.key.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapred.mapoutput.value.class</name>
<value>org.apache.hadoop.io.NullWritable</value>
</property>
<property>
<name>mapred.output.key.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapred.output.value.class</name>
<value>org.apache.hadoop.io.NullWritable</value>
</property>
These classes match with the respective mapper and reducer functions.
I have also tried to change the java mapper and reducer code to try both the old api and the new api. Still the same error.
Regards,
Re: hadoop - oozie versioning issues
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Harshal,
Please look at the following, it seems this is your problem.
Thanks.
Alejandro
---------- Forwarded message ----------
From: Mohammad Islam <ka...@yahoo-inc.com>
Date: Tue, Jul 26, 2011 at 2:10 PM
Subject: Re: [Oozie-users] about the org.apache.hadoop.mapreduce and
org.apache.hadoop.mapred
To: "Oozie-users@yahoogroups.com" <Oo...@yahoogroups.com>
https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
And look for..
*How to run Map-reduce job written using new Hadoop API?
*
*
*
On Thu, Dec 1, 2011 at 7:07 AM, Harshal Vora <ha...@komli.com> wrote:
> **
>
>
> Hey,
>
> We have a map reduce job which is running perfectly well without oozie.
> Now we are trying to integrate it with oozie.
>
> If i set mapred.mapper/reducer.new-api to true, i am getting the following
> exception
>
> java.io.IOException: Type mismatch in key from map: expected
> org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:871)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
> at
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>
>
> And if i comment that part, I am getting the following exception
>
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
> ....
> ....
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class
> net.atomex.amt.segcalc.preprocessor.pixel.mapreduce.PixelLogProcessorMapper
> not org.apache.hadoop.mapred.Mapper
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1028)
> at
> org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:968)
>
>
> I tried to google the exceptions. I learned that for the Error in
> configuring object, we should add the new-api property in the oozie config.
> And for the Type mismatch in key from map, somewhere I read that oozie is
> not compatible with new-api, which i do not think is the case.
>
> I have set mapred.mapoutput/output.key.class, to Text and the value class
> to NullWritable in a config-default.xml as below, and
> config-default is read using <job-xml>.
>
> <property>
> <name>mapred.mapoutput.key.class</name>
> <value>org.apache.hadoop.io.Text</value>
> </property>
> <property>
> <name>mapred.mapoutput.value.class</name>
> <value>org.apache.hadoop.io.NullWritable</value>
> </property>
> <property>
> <name>mapred.output.key.class</name>
> <value>org.apache.hadoop.io.Text</value>
> </property>
> <property>
> <name>mapred.output.value.class</name>
> <value>org.apache.hadoop.io.NullWritable</value>
> </property>
>
>
> These classes match with the respective mapper and reducer functions.
> I have also tried to change the java mapper and reducer code to try both
> the old api and the new api. Still the same error.
>
>
>
> Regards,
>
>