You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Abraham Elmahrek <ab...@cloudera.com> on 2015/07/06 19:51:14 UTC
Re: Review Request 35556: SQOOP-1094: Add avro support to merge tool
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35556/#review90401
-----------------------------------------------------------
Ship it!
Kill excess whitespace!
Also, TestMerge.testTextFileMerge fails for me!
java.lang.Exception: java.lang.NoSuchMethodError: org.apache.sqoop.mapreduce.MergeTextMapper.processRecord(Lcom/cloudera/sqoop/lib/SqoopRecord;Lorg/apache/hadoop/mapreduce/Mapper$Context;)V
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:399)
Caused by: java.lang.NoSuchMethodError: org.apache.sqoop.mapreduce.MergeTextMapper.processRecord(Lcom/cloudera/sqoop/lib/SqoopRecord;Lorg/apache/hadoop/mapreduce/Mapper$Context;)V
at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:58)
at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:34)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:757)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:231)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Maybe it's local to my box?
src/java/org/apache/sqoop/mapreduce/MergeAvroMapper.java (line 35)
<https://reviews.apache.org/r/35556/#comment143591>
Kill import?
src/java/org/apache/sqoop/mapreduce/MergeJob.java (line 128)
<https://reviews.apache.org/r/35556/#comment143447>
Switch statement would work well here since it's an Enum.
src/java/org/apache/sqoop/mapreduce/MergeJob.java (line 169)
<https://reviews.apache.org/r/35556/#comment143592>
Avro Schema:
src/java/org/apache/sqoop/mapreduce/MergeMapperBase.java (line 31)
<https://reviews.apache.org/r/35556/#comment143593>
unnecessary
src/java/org/apache/sqoop/mapreduce/MergeRecord.java (line 28)
<https://reviews.apache.org/r/35556/#comment143594>
unnecessary
- Abraham Elmahrek
On June 17, 2015, 11:55 a.m., Yibing Shi wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35556/
> -----------------------------------------------------------
>
> (Updated June 17, 2015, 11:55 a.m.)
>
>
> Review request for Sqoop and Abraham Elmahrek.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> SQOOP-1094: Add avro support to merge tool
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/mapreduce/AvroJob.java bb4755c880d4b70f812caf5e812135400602ee36
> src/java/org/apache/sqoop/mapreduce/MergeAvroMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MergeAvroReducer.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MergeJob.java 4e2a916911e7f47838366edf46b5ba5073502453
> src/java/org/apache/sqoop/mapreduce/MergeMapperBase.java c4fd9f781a86b0ae3d9e9af14e40e33e0f198fe2
> src/java/org/apache/sqoop/mapreduce/MergeRecord.java d73112c91f0ea951a6a8b89026ddf79546fa50ff
> src/java/org/apache/sqoop/mapreduce/MergeReducer.java cafff8ab0609cd0580ff85e49082a59ce68e7141
> src/java/org/apache/sqoop/mapreduce/MergeReducerBase.java PRE-CREATION
> src/test/com/cloudera/sqoop/TestMerge.java 3821aa1c153d36135920d72666ffcd174e9c0bb8
>
> Diff: https://reviews.apache.org/r/35556/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Yibing Shi
>
>
Re: Review Request 35556: SQOOP-1094: Add avro support to merge tool
Posted by Yibing Shi <sh...@gmail.com>.
> On July 7, 2015, 3:51 a.m., Abraham Elmahrek wrote:
> > Kill excess whitespace!
> >
> > Also, TestMerge.testTextFileMerge fails for me!
> >
> > java.lang.Exception: java.lang.NoSuchMethodError: org.apache.sqoop.mapreduce.MergeTextMapper.processRecord(Lcom/cloudera/sqoop/lib/SqoopRecord;Lorg/apache/hadoop/mapreduce/Mapper$Context;)V
> > at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:399)
> > Caused by: java.lang.NoSuchMethodError: org.apache.sqoop.mapreduce.MergeTextMapper.processRecord(Lcom/cloudera/sqoop/lib/SqoopRecord;Lorg/apache/hadoop/mapreduce/Mapper$Context;)V
> > at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:58)
> > at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:34)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:757)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
> > at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:231)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Maybe it's local to my box?
I cannot reproduce the error. Looking at the source code, we do have the processRecord method in MergeTextMapper (via its parent MergeMapperBase). Could you please help try again?
- Yibing
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35556/#review90401
-----------------------------------------------------------
On June 17, 2015, 9:55 p.m., Yibing Shi wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35556/
> -----------------------------------------------------------
>
> (Updated June 17, 2015, 9:55 p.m.)
>
>
> Review request for Sqoop and Abraham Elmahrek.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> SQOOP-1094: Add avro support to merge tool
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/mapreduce/AvroJob.java bb4755c880d4b70f812caf5e812135400602ee36
> src/java/org/apache/sqoop/mapreduce/MergeAvroMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MergeAvroReducer.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MergeJob.java 4e2a916911e7f47838366edf46b5ba5073502453
> src/java/org/apache/sqoop/mapreduce/MergeMapperBase.java c4fd9f781a86b0ae3d9e9af14e40e33e0f198fe2
> src/java/org/apache/sqoop/mapreduce/MergeRecord.java d73112c91f0ea951a6a8b89026ddf79546fa50ff
> src/java/org/apache/sqoop/mapreduce/MergeReducer.java cafff8ab0609cd0580ff85e49082a59ce68e7141
> src/java/org/apache/sqoop/mapreduce/MergeReducerBase.java PRE-CREATION
> src/test/com/cloudera/sqoop/TestMerge.java 3821aa1c153d36135920d72666ffcd174e9c0bb8
>
> Diff: https://reviews.apache.org/r/35556/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Yibing Shi
>
>