You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Wu YongBo <wu...@mediav.com> on 2013/08/12 15:23:20 UTC
答复: sqoop generate java class in old mapreduce API?
I used classes generated by sqoop as Key-Value, and set OutputFormat to
SequenceFileOutputFormat, and the error occurred.
However, when I simply changed the Key-Value to Text, and OutputFormat to
TextOutputFormat, it worked!
SqoopGenClass value = ....;
context.write(value, NullWritable.get());
=====>>>>
context.write(new Text(value.toString()), NullWritable.get());
Our hadoop version is 0.20.2,that's too old so I gusss this is the cause.
-----邮件原件-----
发件人: Jarek Jarcec Cecho [mailto:jarcec@apache.org]
发送时间: 12 August 2013 08:35
收件人: user@sqoop.apache.org
主题: Re: sqoop generate java class in old mapreduce API?
Hi Yongbo,
I believe that the writable itself should be independent on the new vs. old
mapreduce API. Would you mind sharing more details about your use case?
Maybe a small code example reproducing the issue could help us move forward.
Jarcec
On Thu, Aug 08, 2013 at 09:10:50PM +0800, Wu YongBo wrote:
> Hi All,
>
>
>
> I am using a java class generated by sqoop(1.4.2), and I get the
> following error:
>
>
>
> 2013-08-08 20:43:10,854 WARN org.apache.hadoop.mapred.TaskTracker:
> Error running child
>
> java.lang.ClassCastException: class
> com.a.semv.io.semreportbase_keyword
>
> at java.lang.Class.asSubclass(Class.java:3018)
>
> at
> org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:5
> 99)
>
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:7
> 91)
>
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.jav
> a:524)
>
> at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
>
> My job is written in mapreduce new API.
>
> My hadoop version:0.20.2
>
>
>
> Thanks,
>
> Yongbo
>
>
>
Re: 答复: sqoop generate java class in old mapreduce API?
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Hi sir,
Sqoop generated class implements Hadoop interface called Writable, but do not implements interface WritableComparable. Whereas implementing Writable interface is sufficient for serialization it's not sufficient for keys that are serialized from mappers. Instead interface WritableComparable is needed in such case. My guess is that the generated class is used in a mapper in a job with reduce phase that is violating this requirement. Would you mind sharing a small code example that reproduces the issue? (e.g. multiple java classes that together put whole mapreduce job).
Jarcec
On Mon, Aug 12, 2013 at 09:23:20PM +0800, Wu YongBo wrote:
> I used classes generated by sqoop as Key-Value, and set OutputFormat to
> SequenceFileOutputFormat, and the error occurred.
> However, when I simply changed the Key-Value to Text, and OutputFormat to
> TextOutputFormat, it worked!
>
> SqoopGenClass value = ....;
>
> context.write(value, NullWritable.get());
> =====>>>>
> context.write(new Text(value.toString()), NullWritable.get());
>
> Our hadoop version is 0.20.2,that's too old so I gusss this is the cause.
>
> -----邮件原件-----
> 发件人: Jarek Jarcec Cecho [mailto:jarcec@apache.org]
> 发送时间: 12 August 2013 08:35
> 收件人: user@sqoop.apache.org
> 主题: Re: sqoop generate java class in old mapreduce API?
>
> Hi Yongbo,
> I believe that the writable itself should be independent on the new vs. old
> mapreduce API. Would you mind sharing more details about your use case?
> Maybe a small code example reproducing the issue could help us move forward.
>
> Jarcec
>
> On Thu, Aug 08, 2013 at 09:10:50PM +0800, Wu YongBo wrote:
> > Hi All,
> >
> >
> >
> > I am using a java class generated by sqoop(1.4.2), and I get the
> > following error:
> >
> >
> >
> > 2013-08-08 20:43:10,854 WARN org.apache.hadoop.mapred.TaskTracker:
> > Error running child
> >
> > java.lang.ClassCastException: class
> > com.a.semv.io.semreportbase_keyword
> >
> > at java.lang.Class.asSubclass(Class.java:3018)
> >
> > at
> > org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:5
> > 99)
> >
> > at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:7
> > 91)
> >
> > at
> > org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.jav
> > a:524)
> >
> > at
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> >
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >
> >
> >
> > My job is written in mapreduce new API.
> >
> > My hadoop version:0.20.2
> >
> >
> >
> > Thanks,
> >
> > Yongbo
> >
> >
> >
>