You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Szabolcs Vasas <va...@gmail.com> on 2016/05/31 22:23:52 UTC

Review Request 48097: Sqoop2: Avro support for HdfsConnector

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48097/
-----------------------------------------------------------

Review request for Sqoop.


Bugs: SQOOP-2932
    https://issues.apache.org/jira/browse/SQOOP-2932


Repository: sqoop-sqoop2


Description
-------

Sqoop2: Avro support for HdfsConnector


Diffs
-----

  connector/connector-hdfs/pom.xml 37cf3fa 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/AvroRecordReader.java PRE-CREATION 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java 5973463 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java 7cef93c 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/ToFormat.java ffce583 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/HdfsAvroWriter.java PRE-CREATION 
  connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestLoader.java cbd555a 
  pom.xml 891b2c9 
  test/src/test/java/org/apache/sqoop/integration/connector/hdfs/AvroTest.java PRE-CREATION 
  test/src/test/java/org/apache/sqoop/integration/connector/hdfs/NullValueTest.java 1e8c688 

Diff: https://reviews.apache.org/r/48097/diff/


Testing
-------

Unit test, new integration test is added.


Thanks,

Szabolcs Vasas

Re: Review Request 48097: Sqoop2: Avro support for HdfsConnector

Posted by Abraham Fine <ab...@abrahamfine.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48097/#review136515
-----------------------------------------------------------



- Abraham Fine


On June 4, 2016, 9:54 p.m., Szabolcs Vasas wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48097/
> -----------------------------------------------------------
> 
> (Updated June 4, 2016, 9:54 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2932
>     https://issues.apache.org/jira/browse/SQOOP-2932
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> Sqoop2: Avro support for HdfsConnector
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java 7488ae9 
>   connector/connector-hdfs/pom.xml 37cf3fa 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java 5973463 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java 7cef93c 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/ToFormat.java ffce583 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/HdfsAvroWriter.java PRE-CREATION 
>   connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestLoader.java cbd555a 
>   pom.xml 891b2c9 
>   test/src/test/java/org/apache/sqoop/integration/connector/hdfs/AvroTest.java PRE-CREATION 
>   test/src/test/java/org/apache/sqoop/integration/connector/hdfs/NullValueTest.java 1e8c688 
> 
> Diff: https://reviews.apache.org/r/48097/diff/
> 
> 
> Testing
> -------
> 
> Unit test, new integration test is added.
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 48097: Sqoop2: Avro support for HdfsConnector

Posted by Abraham Fine <ab...@abrahamfine.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48097/#review135813
-----------------------------------------------------------




connector/connector-hdfs/pom.xml (line 86)
<https://reviews.apache.org/r/48097/#comment200857>

    why do we need this?



connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/AvroRecordReader.java (line 38)
<https://reviews.apache.org/r/48097/#comment200856>

    doesn't the avro api already have one of these?



connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java (line 29)
<https://reviews.apache.org/r/48097/#comment200858>

    we prefer not to use * imports.



connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java (line 237)
<https://reviews.apache.org/r/48097/#comment200855>

    can we somehow reduce code between this method and the extractParquetFile method?



connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java (line 327)
<https://reviews.apache.org/r/48097/#comment200859>

    is there a more specific exception?



test/src/test/java/org/apache/sqoop/integration/connector/hdfs/AvroTest.java (line 127)
<https://reviews.apache.org/r/48097/#comment201578>

    can we extract this logic so it can be shared with other tests.


- Abraham Fine


On June 4, 2016, 9:54 p.m., Szabolcs Vasas wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48097/
> -----------------------------------------------------------
> 
> (Updated June 4, 2016, 9:54 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2932
>     https://issues.apache.org/jira/browse/SQOOP-2932
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> Sqoop2: Avro support for HdfsConnector
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java 7488ae9 
>   connector/connector-hdfs/pom.xml 37cf3fa 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java 5973463 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java 7cef93c 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/ToFormat.java ffce583 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/HdfsAvroWriter.java PRE-CREATION 
>   connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestLoader.java cbd555a 
>   pom.xml 891b2c9 
>   test/src/test/java/org/apache/sqoop/integration/connector/hdfs/AvroTest.java PRE-CREATION 
>   test/src/test/java/org/apache/sqoop/integration/connector/hdfs/NullValueTest.java 1e8c688 
> 
> Diff: https://reviews.apache.org/r/48097/diff/
> 
> 
> Testing
> -------
> 
> Unit test, new integration test is added.
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 48097: Sqoop2: Avro support for HdfsConnector

Posted by Szabolcs Vasas <va...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48097/
-----------------------------------------------------------

(Updated June 16, 2016, 3:11 p.m.)


Review request for Sqoop.


Bugs: SQOOP-2932
    https://issues.apache.org/jira/browse/SQOOP-2932


Repository: sqoop-sqoop2


Description
-------

Sqoop2: Avro support for HdfsConnector


Diffs (updated)
-----

  common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java 7488ae9 
  connector/connector-hdfs/pom.xml 37cf3fa 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java 5973463 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java 7cef93c 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/ToFormat.java ffce583 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/HdfsAvroWriter.java PRE-CREATION 
  connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestLoader.java cbd555a 
  pom.xml 891b2c9 
  test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 913cdbe 
  test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 0369994 
  test/src/test/java/org/apache/sqoop/integration/connector/hdfs/AvroTest.java PRE-CREATION 
  test/src/test/java/org/apache/sqoop/integration/connector/hdfs/BinaryFileFormatTestBase.java PRE-CREATION 
  test/src/test/java/org/apache/sqoop/integration/connector/hdfs/NullValueTest.java 1e8c688 
  test/src/test/java/org/apache/sqoop/integration/connector/hdfs/ParquetTest.java d55563d 

Diff: https://reviews.apache.org/r/48097/diff/


Testing
-------

Unit test, new integration test is added.


Thanks,

Szabolcs Vasas

Re: Review Request 48097: Sqoop2: Avro support for HdfsConnector

Posted by Szabolcs Vasas <va...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48097/
-----------------------------------------------------------

(Updated June 4, 2016, 9:54 p.m.)


Review request for Sqoop.


Bugs: SQOOP-2932
    https://issues.apache.org/jira/browse/SQOOP-2932


Repository: sqoop-sqoop2


Description
-------

Sqoop2: Avro support for HdfsConnector


Diffs (updated)
-----

  common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java 7488ae9 
  connector/connector-hdfs/pom.xml 37cf3fa 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java 5973463 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java 7cef93c 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/ToFormat.java ffce583 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/HdfsAvroWriter.java PRE-CREATION 
  connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestLoader.java cbd555a 
  pom.xml 891b2c9 
  test/src/test/java/org/apache/sqoop/integration/connector/hdfs/AvroTest.java PRE-CREATION 
  test/src/test/java/org/apache/sqoop/integration/connector/hdfs/NullValueTest.java 1e8c688 

Diff: https://reviews.apache.org/r/48097/diff/


Testing
-------

Unit test, new integration test is added.


Thanks,

Szabolcs Vasas

Re: Review Request 48097: Sqoop2: Avro support for HdfsConnector

Posted by Szabolcs Vasas <va...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48097/
-----------------------------------------------------------

(Updated June 4, 2016, 7:51 a.m.)


Review request for Sqoop.


Bugs: SQOOP-2932
    https://issues.apache.org/jira/browse/SQOOP-2932


Repository: sqoop-sqoop2


Description
-------

Sqoop2: Avro support for HdfsConnector


Diffs (updated)
-----

  common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java 7488ae9 
  connector/connector-hdfs/pom.xml 37cf3fa 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java 5973463 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java 7cef93c 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/ToFormat.java ffce583 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/HdfsAvroWriter.java PRE-CREATION 
  connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestLoader.java cbd555a 
  pom.xml 891b2c9 
  test/src/test/java/org/apache/sqoop/integration/connector/hdfs/AvroTest.java PRE-CREATION 
  test/src/test/java/org/apache/sqoop/integration/connector/hdfs/NullValueTest.java 1e8c688 

Diff: https://reviews.apache.org/r/48097/diff/


Testing
-------

Unit test, new integration test is added.


Thanks,

Szabolcs Vasas