You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Qian Xu <sx...@googlemail.com> on 2014/12/16 08:02:11 UTC
Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/
-----------------------------------------------------------
Review request for Sqoop.
Bugs: SQOOP-1647
https://issues.apache.org/jira/browse/SQOOP-1647
Repository: sqoop-sqoop2
Description
-------
This JIRA proposes to support reading data from an HDFS dataset.
Here are two notes:
1. Due to a review board warning, I moved new code from KiteDataTypeUtil.java to SqoopDataTypeUtil.
2. Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
Diffs
-----
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882abf352236a5142608c31b7180e5ae9101
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de3a65f83f50e94b54aff43a169aa4dbd2f
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4bf41dcd691f64546b18c7ede029b57e339
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 709fd94578eff5a982e0bcbd0aa2065958a7105f
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b496af9a71211177a5de3b45308c71389be
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b468753cca291a62f76cb28fb1a12e2f051
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5084c526fd4f4da534e9a1f603417ce77d
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
Diff: https://reviews.apache.org/r/29090/diff/
Testing
-------
Added new test cases.
Thanks,
Qian Xu
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Qian Xu <qi...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/
-----------------------------------------------------------
(Updated Jan. 8, 2015, 11:45 a.m.)
Review request for Sqoop.
Changes
-------
Fixed a merge conflict
Bugs: SQOOP-1647
https://issues.apache.org/jira/browse/SQOOP-1647
Repository: sqoop-sqoop2
Description
-------
This JIRA proposes to support reading data from an HDFS dataset.
Note that Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
Diffs (updated)
-----
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java 982d6dd
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4b
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java b115242
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 3b36f1d
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java ad5898f
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
connector/connector-kite/src/main/resources/kite-connector-config.properties 65541c5
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToDestroyer.java 87ed906
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToInitializer.java fab31f9
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/AvroDataTypeUtil.java PRE-CREATION
Diff: https://reviews.apache.org/r/29090/diff/
Testing
-------
Added new test cases.
Thanks,
Qian Xu
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Qian Xu <qi...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/
-----------------------------------------------------------
(Updated Jan. 8, 2015, 11:37 a.m.)
Review request for Sqoop.
Changes
-------
Rebased to current code base
Bugs: SQOOP-1647
https://issues.apache.org/jira/browse/SQOOP-1647
Repository: sqoop-sqoop2
Description
-------
This JIRA proposes to support reading data from an HDFS dataset.
Note that Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
Diffs (updated)
-----
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java 982d6dd
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4b
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java b115242
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 3b36f1d
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java ad5898f
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
connector/connector-kite/src/main/resources/kite-connector-config.properties 65541c5
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToDestroyer.java 87ed906
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToInitializer.java fab31f9
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/AvroDataTypeUtil.java PRE-CREATION
pom.xml d1f0a45
Diff: https://reviews.apache.org/r/29090/diff/
Testing
-------
Added new test cases.
Thanks,
Qian Xu
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/#review66138
-----------------------------------------------------------
Ship it!
my comments have been addressed. thanks Qian, nice work.!
- Veena Basavaraj
On Dec. 23, 2014, 5:23 p.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29090/
> -----------------------------------------------------------
>
> (Updated Dec. 23, 2014, 5:23 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1647
> https://issues.apache.org/jira/browse/SQOOP-1647
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> This JIRA proposes to support reading data from an HDFS dataset.
>
> Note that Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
>
>
> Diffs
> -----
>
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4b
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 0a46f4a
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b4
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 11233a8
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
> connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b4
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToDestroyer.java 4051fda
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToInitializer.java 5f0525d
> connector/connector-sdk/pom.xml 38c217a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/AvroDataTypeUtil.java PRE-CREATION
> pom.xml afdeb75
>
> Diff: https://reviews.apache.org/r/29090/diff/
>
>
> Testing
> -------
>
> Added new test cases.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/#review66545
-----------------------------------------------------------
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/AvroDataTypeUtil.java
<https://reviews.apache.org/r/29090/#comment110117>
this class can be combined into one SqoopAvroUtils, that I am planning to address as part of SQOOP-1616
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/AvroDataTypeUtil.java
<https://reviews.apache.org/r/29090/#comment110115>
please call this avroToSqoop or something like it that makes it clear that this is converting from avro to sqoop
- Veena Basavaraj
On Dec. 23, 2014, 5:23 p.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29090/
> -----------------------------------------------------------
>
> (Updated Dec. 23, 2014, 5:23 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1647
> https://issues.apache.org/jira/browse/SQOOP-1647
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> This JIRA proposes to support reading data from an HDFS dataset.
>
> Note that Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
>
>
> Diffs
> -----
>
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4b
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 0a46f4a
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b4
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 11233a8
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
> connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b4
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToDestroyer.java 4051fda
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToInitializer.java 5f0525d
> connector/connector-sdk/pom.xml 38c217a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/AvroDataTypeUtil.java PRE-CREATION
> pom.xml afdeb75
>
> Diff: https://reviews.apache.org/r/29090/diff/
>
>
> Testing
> -------
>
> Added new test cases.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/#review67155
-----------------------------------------------------------
Ship it!
Qian, LGTM.
- Abraham Elmahrek
On Dec. 24, 2014, 1:23 a.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29090/
> -----------------------------------------------------------
>
> (Updated Dec. 24, 2014, 1:23 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1647
> https://issues.apache.org/jira/browse/SQOOP-1647
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> This JIRA proposes to support reading data from an HDFS dataset.
>
> Note that Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
>
>
> Diffs
> -----
>
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4b
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 0a46f4a
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b4
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 11233a8
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
> connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b4
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToDestroyer.java 4051fda
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToInitializer.java 5f0525d
> connector/connector-sdk/pom.xml 38c217a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/AvroDataTypeUtil.java PRE-CREATION
> pom.xml afdeb75
>
> Diff: https://reviews.apache.org/r/29090/diff/
>
>
> Testing
> -------
>
> Added new test cases.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/
-----------------------------------------------------------
(Updated Dec. 24, 2014, 9:23 a.m.)
Review request for Sqoop.
Bugs: SQOOP-1647
https://issues.apache.org/jira/browse/SQOOP-1647
Repository: sqoop-sqoop2
Description (updated)
-------
This JIRA proposes to support reading data from an HDFS dataset.
Note that Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
Diffs
-----
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4b
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 0a46f4a
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b4
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 11233a8
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b4
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToDestroyer.java 4051fda
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToInitializer.java 5f0525d
connector/connector-sdk/pom.xml 38c217a
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/AvroDataTypeUtil.java PRE-CREATION
pom.xml afdeb75
Diff: https://reviews.apache.org/r/29090/diff/
Testing
-------
Added new test cases.
Thanks,
Qian Xu
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/
-----------------------------------------------------------
(Updated Dec. 23, 2014, 3:02 p.m.)
Review request for Sqoop.
Changes
-------
Changed code regarding comments of Veena and Ryan.
Bugs: SQOOP-1647
https://issues.apache.org/jira/browse/SQOOP-1647
Repository: sqoop-sqoop2
Description
-------
This JIRA proposes to support reading data from an HDFS dataset.
Here are two notes:
1. Due to a review board warning, I moved new code from KiteDataTypeUtil.java to AvroDataTypeUtil.
2. Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
Diffs (updated)
-----
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4b
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 0a46f4a
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b4
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 11233a8
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b4
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToDestroyer.java 4051fda
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToInitializer.java 5f0525d
connector/connector-sdk/pom.xml 38c217a
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/AvroDataTypeUtil.java PRE-CREATION
pom.xml afdeb75
Diff: https://reviews.apache.org/r/29090/diff/
Testing
-------
Added new test cases.
Thanks,
Qian Xu
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/#review65687
-----------------------------------------------------------
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java
<https://reviews.apache.org/r/29090/#comment108898>
Also another thing I have to note is, this is standard practice in sqoop that I have not seen lesewhere, where everything is wrapped in a sqoop exception and exception is sent in JSON.! instead of a error code.
- Veena Basavaraj
On Dec. 16, 2014, 12:58 a.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29090/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2014, 12:58 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1647
> https://issues.apache.org/jira/browse/SQOOP-1647
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> This JIRA proposes to support reading data from an HDFS dataset.
>
> Here are two notes:
> 1. Due to a review board warning, I moved new code from KiteDataTypeUtil.java to AvroDataTypeUtil.
> 2. Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
>
>
> Diffs
> -----
>
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882abf352236a5142608c31b7180e5ae9101
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de3a65f83f50e94b54aff43a169aa4dbd2f
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4bf41dcd691f64546b18c7ede029b57e339
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 709fd94578eff5a982e0bcbd0aa2065958a7105f
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b496af9a71211177a5de3b45308c71389be
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/util/AvroDataTypeUtil.java PRE-CREATION
> connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b468753cca291a62f76cb28fb1a12e2f051
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5084c526fd4f4da534e9a1f603417ce77d
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/29090/diff/
>
>
> Testing
> -------
>
> Added new test cases.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Qian Xu <sx...@googlemail.com>.
> On Dec. 17, 2014, 2:53 a.m., Veena Basavaraj wrote:
> > connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java, line 61
> > <https://reviews.apache.org/r/29090/diff/2/?file=792928#file792928line61>
> >
> > we need to convert it to the write object types for date, which version of avro are we suing, does it not support date types yet?
> >
> > see an example in GenericJDBCConnector
Will fix in SQOOP-1616
> On Dec. 17, 2014, 2:53 a.m., Veena Basavaraj wrote:
> > connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java, line 84
> > <https://reviews.apache.org/r/29090/diff/2/?file=792938#file792938line84>
> >
> > can we also test cases for different data types, so how does it handle date-time or array? if it is writing it as object array are we making sure it passes the actual IDF formats for objects?
> > https://cwiki.apache.org/confluence/display/SQOOP/Intermediate+Data+Format+API#IntermediateDataFormatAPI-1.99.5SQOOPCSVandObjectFormat
Will fix in SQOOP-1616
- Qian
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/#review65227
-----------------------------------------------------------
On Dec. 23, 2014, 3:02 p.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29090/
> -----------------------------------------------------------
>
> (Updated Dec. 23, 2014, 3:02 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1647
> https://issues.apache.org/jira/browse/SQOOP-1647
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> This JIRA proposes to support reading data from an HDFS dataset.
>
> Here are two notes:
> 1. Due to a review board warning, I moved new code from KiteDataTypeUtil.java to AvroDataTypeUtil.
> 2. Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
>
>
> Diffs
> -----
>
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4b
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 0a46f4a
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b4
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 11233a8
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
> connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b4
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToDestroyer.java 4051fda
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToInitializer.java 5f0525d
> connector/connector-sdk/pom.xml 38c217a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/AvroDataTypeUtil.java PRE-CREATION
> pom.xml afdeb75
>
> Diff: https://reviews.apache.org/r/29090/diff/
>
>
> Testing
> -------
>
> Added new test cases.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/#review65227
-----------------------------------------------------------
good start!
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java
<https://reviews.apache.org/r/29090/#comment108261>
we need to convert it to the write object types for date, which version of avro are we suing, does it not support date types yet?
see an example in GenericJDBCConnector
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/util/AvroDataTypeUtil.java
<https://reviews.apache.org/r/29090/#comment108259>
may be worth moving this to connector-sdk, since I am in the process of adding a util as part of AVroIDF support SQOOP-1705
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java
<https://reviews.apache.org/r/29090/#comment108258>
can we also test cases for different data types, so how does it handle date-time or array? if it is writing it as object array are we making sure it passes the actual IDF formats for objects?
https://cwiki.apache.org/confluence/display/SQOOP/Intermediate+Data+Format+API#IntermediateDataFormatAPI-1.99.5SQOOPCSVandObjectFormat
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java
<https://reviews.apache.org/r/29090/#comment108257>
love your tests man! you must have max coverage of all the sqoop packages
good s
- Veena Basavaraj
On Dec. 16, 2014, 12:58 a.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29090/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2014, 12:58 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1647
> https://issues.apache.org/jira/browse/SQOOP-1647
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> This JIRA proposes to support reading data from an HDFS dataset.
>
> Here are two notes:
> 1. Due to a review board warning, I moved new code from KiteDataTypeUtil.java to AvroDataTypeUtil.
> 2. Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
>
>
> Diffs
> -----
>
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882abf352236a5142608c31b7180e5ae9101
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de3a65f83f50e94b54aff43a169aa4dbd2f
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4bf41dcd691f64546b18c7ede029b57e339
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 709fd94578eff5a982e0bcbd0aa2065958a7105f
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b496af9a71211177a5de3b45308c71389be
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/util/AvroDataTypeUtil.java PRE-CREATION
> connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b468753cca291a62f76cb28fb1a12e2f051
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5084c526fd4f4da534e9a1f603417ce77d
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/29090/diff/
>
>
> Testing
> -------
>
> Added new test cases.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Veena Basavaraj <vb...@cloudera.com>.
> On Dec. 19, 2014, 1:17 p.m., Ryan Blue wrote:
> > connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java, line 69
> > <https://reviews.apache.org/r/29090/diff/2/?file=792925#file792925line69>
> >
> > @VisibleForTesting thing should be package-private instead of protected, right? Protected leaks the method to children outside of this package, while package-private only allows access from within this package.
agree with Ryan, we have not used this much acrss the entire code and there is lot of these places where we have made things public and protected
> On Dec. 19, 2014, 1:17 p.m., Ryan Blue wrote:
> > connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java, line 50
> > <https://reviews.apache.org/r/29090/diff/2/?file=792927#file792927line50>
> >
> > Why is this wrapped in a try/catch? Exception is always too general to catch. If there are no checked exceptions but there is still fear of failure, it should catch RuntimeException.
very good catch Ryan, i missed it
- Veena
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/#review65564
-----------------------------------------------------------
On Dec. 16, 2014, 12:58 a.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29090/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2014, 12:58 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1647
> https://issues.apache.org/jira/browse/SQOOP-1647
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> This JIRA proposes to support reading data from an HDFS dataset.
>
> Here are two notes:
> 1. Due to a review board warning, I moved new code from KiteDataTypeUtil.java to AvroDataTypeUtil.
> 2. Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
>
>
> Diffs
> -----
>
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882abf352236a5142608c31b7180e5ae9101
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de3a65f83f50e94b54aff43a169aa4dbd2f
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4bf41dcd691f64546b18c7ede029b57e339
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 709fd94578eff5a982e0bcbd0aa2065958a7105f
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b496af9a71211177a5de3b45308c71389be
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/util/AvroDataTypeUtil.java PRE-CREATION
> connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b468753cca291a62f76cb28fb1a12e2f051
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5084c526fd4f4da534e9a1f603417ce77d
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/29090/diff/
>
>
> Testing
> -------
>
> Added new test cases.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Qian Xu <sx...@googlemail.com>.
> On Dec. 20, 2014, 5:17 a.m., Ryan Blue wrote:
> > connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java, line 43
> > <https://reviews.apache.org/r/29090/diff/2/?file=792927#file792927line43>
> >
> > This is right that it isn't any more efficient to partition an un-partitioned dataset. But a partitioned dataset could return partitions. We can implement it later.
Will fix in SQOOP-1942
> On Dec. 20, 2014, 5:17 a.m., Ryan Blue wrote:
> > connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java, line 46
> > <https://reviews.apache.org/r/29090/diff/2/?file=792930#file792930line46>
> >
> > Why doesn't this call `Datasets.exists` directly?
At that time I planed to wrap kite sdk usage in `KiteDatasetExecutor`. But I think it is over-designed.
> On Dec. 20, 2014, 5:17 a.m., Ryan Blue wrote:
> > connector/connector-kite/src/main/resources/kite-connector-config.properties, line 38
> > <https://reviews.apache.org/r/29090/diff/2/?file=792936#file792936line38>
> >
> > Without "store", this no longer makes sense. I would change it to "Dataset URI" because that could be used for both from and to URIs.
I've added "store" and "load" back. We have `toJobConfig.uri` and `fromJobConfig.uri`.
- Qian
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/#review65564
-----------------------------------------------------------
On Dec. 23, 2014, 3:02 p.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29090/
> -----------------------------------------------------------
>
> (Updated Dec. 23, 2014, 3:02 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1647
> https://issues.apache.org/jira/browse/SQOOP-1647
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> This JIRA proposes to support reading data from an HDFS dataset.
>
> Here are two notes:
> 1. Due to a review board warning, I moved new code from KiteDataTypeUtil.java to AvroDataTypeUtil.
> 2. Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
>
>
> Diffs
> -----
>
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4b
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 0a46f4a
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b4
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 11233a8
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
> connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b4
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToDestroyer.java 4051fda
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteToInitializer.java 5f0525d
> connector/connector-sdk/pom.xml 38c217a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/AvroDataTypeUtil.java PRE-CREATION
> pom.xml afdeb75
>
> Diff: https://reviews.apache.org/r/29090/diff/
>
>
> Testing
> -------
>
> Added new test cases.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Ryan Blue <bl...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/#review65564
-----------------------------------------------------------
Looks pretty close overall. Just a few comments.
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java
<https://reviews.apache.org/r/29090/#comment108791>
I think this should be "Error occurred while creating partitions"
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java
<https://reviews.apache.org/r/29090/#comment108792>
@VisibleForTesting thing should be package-private instead of protected, right? Protected leaks the method to children outside of this package, while package-private only allows access from within this package.
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java
<https://reviews.apache.org/r/29090/#comment108889>
This is right that it isn't any more efficient to partition an un-partitioned dataset. But a partitioned dataset could return partitions. We can implement it later.
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java
<https://reviews.apache.org/r/29090/#comment108884>
Why is this wrapped in a try/catch? Exception is always too general to catch. If there are no checked exceptions but there is still fear of failure, it should catch RuntimeException.
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java
<https://reviews.apache.org/r/29090/#comment108886>
Also should be package-private?
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java
<https://reviews.apache.org/r/29090/#comment108890>
Why doesn't this call `Datasets.exists` directly?
connector/connector-kite/src/main/resources/kite-connector-config.properties
<https://reviews.apache.org/r/29090/#comment108794>
Without "store", this no longer makes sense. I would change it to "Dataset URI" because that could be used for both from and to URIs.
- Ryan Blue
On Dec. 16, 2014, 12:58 a.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29090/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2014, 12:58 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1647
> https://issues.apache.org/jira/browse/SQOOP-1647
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> This JIRA proposes to support reading data from an HDFS dataset.
>
> Here are two notes:
> 1. Due to a review board warning, I moved new code from KiteDataTypeUtil.java to AvroDataTypeUtil.
> 2. Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
>
>
> Diffs
> -----
>
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882abf352236a5142608c31b7180e5ae9101
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de3a65f83f50e94b54aff43a169aa4dbd2f
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4bf41dcd691f64546b18c7ede029b57e339
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 709fd94578eff5a982e0bcbd0aa2065958a7105f
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b496af9a71211177a5de3b45308c71389be
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/util/AvroDataTypeUtil.java PRE-CREATION
> connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b468753cca291a62f76cb28fb1a12e2f051
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5084c526fd4f4da534e9a1f603417ce77d
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
> connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/29090/diff/
>
>
> Testing
> -------
>
> Added new test cases.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/
-----------------------------------------------------------
(Updated Dec. 16, 2014, 4:58 p.m.)
Review request for Sqoop.
Changes
-------
Added a missing file and rename it to AvroDataTypeUtil. I don't know exactly why I cannot modify KiteDataTypeUtil.
Bugs: SQOOP-1647
https://issues.apache.org/jira/browse/SQOOP-1647
Repository: sqoop-sqoop2
Description (updated)
-------
This JIRA proposes to support reading data from an HDFS dataset.
Here are two notes:
1. Due to a review board warning, I moved new code from KiteDataTypeUtil.java to AvroDataTypeUtil.
2. Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
Diffs
-----
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882abf352236a5142608c31b7180e5ae9101
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de3a65f83f50e94b54aff43a169aa4dbd2f
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4bf41dcd691f64546b18c7ede029b57e339
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 709fd94578eff5a982e0bcbd0aa2065958a7105f
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b496af9a71211177a5de3b45308c71389be
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/util/AvroDataTypeUtil.java PRE-CREATION
connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b468753cca291a62f76cb28fb1a12e2f051
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5084c526fd4f4da534e9a1f603417ce77d
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
Diff: https://reviews.apache.org/r/29090/diff/
Testing
-------
Added new test cases.
Thanks,
Qian Xu
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/
-----------------------------------------------------------
(Updated Dec. 16, 2014, 4:56 p.m.)
Review request for Sqoop.
Bugs: SQOOP-1647
https://issues.apache.org/jira/browse/SQOOP-1647
Repository: sqoop-sqoop2
Description
-------
This JIRA proposes to support reading data from an HDFS dataset.
Here are two notes:
1. Due to a review board warning, I moved new code from KiteDataTypeUtil.java to SqoopDataTypeUtil.
2. Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
Diffs (updated)
-----
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882abf352236a5142608c31b7180e5ae9101
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de3a65f83f50e94b54aff43a169aa4dbd2f
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4bf41dcd691f64546b18c7ede029b57e339
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 709fd94578eff5a982e0bcbd0aa2065958a7105f
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b496af9a71211177a5de3b45308c71389be
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/util/AvroDataTypeUtil.java PRE-CREATION
connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b468753cca291a62f76cb28fb1a12e2f051
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5084c526fd4f4da534e9a1f603417ce77d
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
Diff: https://reviews.apache.org/r/29090/diff/
Testing
-------
Added new test cases.
Thanks,
Qian Xu
Re: Review Request 29090: SQOOP-1647: KiteConnector FROM-side HDFS
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29090/
-----------------------------------------------------------
(Updated Dec. 16, 2014, 4:49 p.m.)
Review request for Sqoop.
Bugs: SQOOP-1647
https://issues.apache.org/jira/browse/SQOOP-1647
Repository: sqoop-sqoop2
Description
-------
This JIRA proposes to support reading data from an HDFS dataset.
Here are two notes:
1. Due to a review board warning, I moved new code from KiteDataTypeUtil.java to SqoopDataTypeUtil.
2. Kite SDK seems have limitation of opening multiple record reader instances by Sqoop 2's demand. This version supports only one partition.https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/cdk-dev/2NJUvh0WmoY
Diffs
-----
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java c864882abf352236a5142608c31b7180e5ae9101
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnectorError.java d67c8de3a65f83f50e94b54aff43a169aa4dbd2f
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetExecutor.java 9432e4bf41dcd691f64546b18c7ede029b57e339
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartition.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteDatasetPartitioner.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteExtractor.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromDestroyer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteLoader.java 709fd94578eff5a982e0bcbd0aa2065958a7105f
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToDestroyer.java 25912b496af9a71211177a5de3b45308c71389be
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfig.java PRE-CREATION
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/configuration/FromJobConfiguration.java PRE-CREATION
connector/connector-kite/src/main/resources/kite-connector-config.properties 27c77b468753cca291a62f76cb28fb1a12e2f051
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExecutor.java 5e4edc5084c526fd4f4da534e9a1f603417ce77d
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteExtractor.java PRE-CREATION
connector/connector-kite/src/test/java/org/apache/sqoop/connector/kite/TestKiteFromInitializer.java PRE-CREATION
Diff: https://reviews.apache.org/r/29090/diff/
Testing
-------
Added new test cases.
Thanks,
Qian Xu