You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Jarek Cecho <ja...@apache.org> on 2015/03/08 05:23:06 UTC
Review Request 31831: SQOOP-1805 GenericJdbcConnector:
FromJobConfiguration
and Extractor changes if any to support delta read
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31831/
-----------------------------------------------------------
Review request for Sqoop.
Bugs: SQOOP-1805
https://issues.apache.org/jira/browse/SQOOP-1805
Repository: sqoop-sqoop2
Description
-------
I've added incremental options similar to Sqoop 1 and alter our partitioning function to take advantage of them. The method was relatively hard to change, so I've ended up rewriting it completely.
The connector is not persisting the last values back to configuration object as the functionality is not exposed by Sqoop yet.
Diffs
-----
common/src/main/java/org/apache/sqoop/error/code/GenericJdbcConnectorError.java 03bc104
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java 4369071
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExecutor.java 7a01992
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcFromInitializer.java 1ecd152
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java 39e8edd
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/IncrementalImport.java PRE-CREATION
connector/connector-generic-jdbc/src/main/resources/generic-jdbc-connector-config.properties 6a2159b
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestFromInitializer.java 52003ab
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestGenericJdbcConnector.java PRE-CREATION
test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/IncrementalImportTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/31831/diff/
Testing
-------
I've added several new unit tests and integration test for the newly added functionality.
Thanks,
Jarek Cecho
Re: Review Request 31831: SQOOP-1805 GenericJdbcConnector:
FromJobConfiguration and Extractor changes if any to support delta read
Posted by Veena Basavaraj <vy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31831/#review75846
-----------------------------------------------------------
lgtm.
- Veena Basavaraj
On March 7, 2015, 8:23 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31831/
> -----------------------------------------------------------
>
> (Updated March 7, 2015, 8:23 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1805
> https://issues.apache.org/jira/browse/SQOOP-1805
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I've added incremental options similar to Sqoop 1 and alter our partitioning function to take advantage of them. The method was relatively hard to change, so I've ended up rewriting it completely.
>
> The connector is not persisting the last values back to configuration object as the functionality is not exposed by Sqoop yet.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/sqoop/error/code/GenericJdbcConnectorError.java 03bc104
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java 4369071
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExecutor.java 7a01992
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcFromInitializer.java 1ecd152
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java 39e8edd
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/IncrementalImport.java PRE-CREATION
> connector/connector-generic-jdbc/src/main/resources/generic-jdbc-connector-config.properties 6a2159b
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestFromInitializer.java 52003ab
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestGenericJdbcConnector.java PRE-CREATION
> test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/IncrementalImportTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/31831/diff/
>
>
> Testing
> -------
>
> I've added several new unit tests and integration test for the newly added functionality.
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 31831: SQOOP-1805 GenericJdbcConnector:
FromJobConfiguration and Extractor changes if any to support delta read
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31831/#review75673
-----------------------------------------------------------
Ship it!
+1
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcFromInitializer.java
<https://reviews.apache.org/r/31831/#comment122899>
StringUtils.isBlank
- Abraham Elmahrek
On March 8, 2015, 4:23 a.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31831/
> -----------------------------------------------------------
>
> (Updated March 8, 2015, 4:23 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1805
> https://issues.apache.org/jira/browse/SQOOP-1805
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I've added incremental options similar to Sqoop 1 and alter our partitioning function to take advantage of them. The method was relatively hard to change, so I've ended up rewriting it completely.
>
> The connector is not persisting the last values back to configuration object as the functionality is not exposed by Sqoop yet.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/sqoop/error/code/GenericJdbcConnectorError.java 03bc104
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java 4369071
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExecutor.java 7a01992
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcFromInitializer.java 1ecd152
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java 39e8edd
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/IncrementalImport.java PRE-CREATION
> connector/connector-generic-jdbc/src/main/resources/generic-jdbc-connector-config.properties 6a2159b
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestFromInitializer.java 52003ab
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestGenericJdbcConnector.java PRE-CREATION
> test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/IncrementalImportTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/31831/diff/
>
>
> Testing
> -------
>
> I've added several new unit tests and integration test for the newly added functionality.
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 31831: SQOOP-1805 GenericJdbcConnector:
FromJobConfiguration and Extractor changes if any to support delta read
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31831/#review76576
-----------------------------------------------------------
Ship it!
Ship It!
- Abraham Elmahrek
On March 16, 2015, 3:59 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31831/
> -----------------------------------------------------------
>
> (Updated March 16, 2015, 3:59 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1805
> https://issues.apache.org/jira/browse/SQOOP-1805
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I've added incremental options similar to Sqoop 1 and alter our partitioning function to take advantage of them. The method was relatively hard to change, so I've ended up rewriting it completely.
>
> The connector is not persisting the last values back to configuration object as the functionality is not exposed by Sqoop yet.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/sqoop/error/code/GenericJdbcConnectorError.java 03bc104
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java 4369071
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExecutor.java 7a01992
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcFromInitializer.java 1ecd152
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java 39e8edd
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/IncrementalRead.java PRE-CREATION
> connector/connector-generic-jdbc/src/main/resources/generic-jdbc-connector-config.properties 6a2159b
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestFromInitializer.java 52003ab
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestGenericJdbcConnector.java PRE-CREATION
> test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/IncrementalReadTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/31831/diff/
>
>
> Testing
> -------
>
> I've added several new unit tests and integration test for the newly added functionality.
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 31831: SQOOP-1805 GenericJdbcConnector:
FromJobConfiguration and Extractor changes if any to support delta read
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31831/
-----------------------------------------------------------
(Updated March 16, 2015, 3:59 p.m.)
Review request for Sqoop.
Changes
-------
I've incorporated feedback from reviewers:
* Simplified one check
* Renamed incremental "import" to incremental "read"
Bugs: SQOOP-1805
https://issues.apache.org/jira/browse/SQOOP-1805
Repository: sqoop-sqoop2
Description
-------
I've added incremental options similar to Sqoop 1 and alter our partitioning function to take advantage of them. The method was relatively hard to change, so I've ended up rewriting it completely.
The connector is not persisting the last values back to configuration object as the functionality is not exposed by Sqoop yet.
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/error/code/GenericJdbcConnectorError.java 03bc104
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java 4369071
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExecutor.java 7a01992
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcFromInitializer.java 1ecd152
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java 39e8edd
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/IncrementalRead.java PRE-CREATION
connector/connector-generic-jdbc/src/main/resources/generic-jdbc-connector-config.properties 6a2159b
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestFromInitializer.java 52003ab
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestGenericJdbcConnector.java PRE-CREATION
test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/IncrementalReadTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/31831/diff/
Testing
-------
I've added several new unit tests and integration test for the newly added functionality.
Thanks,
Jarek Cecho
Re: Review Request 31831: SQOOP-1805 GenericJdbcConnector:
FromJobConfiguration and Extractor changes if any to support delta read
Posted by Veena Basavaraj <vy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31831/#review76391
-----------------------------------------------------------
connector/connector-generic-jdbc/src/main/resources/generic-jdbc-connector-config.properties
<https://reviews.apache.org/r/31831/#comment123941>
can "import" be replaced with a more generic "read" or "fetch"? we probably want ot move away from IMPORT / EXPORT terms since the FROM/ TO refactoring
- Veena Basavaraj
On March 7, 2015, 8:23 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31831/
> -----------------------------------------------------------
>
> (Updated March 7, 2015, 8:23 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1805
> https://issues.apache.org/jira/browse/SQOOP-1805
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I've added incremental options similar to Sqoop 1 and alter our partitioning function to take advantage of them. The method was relatively hard to change, so I've ended up rewriting it completely.
>
> The connector is not persisting the last values back to configuration object as the functionality is not exposed by Sqoop yet.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/sqoop/error/code/GenericJdbcConnectorError.java 03bc104
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java 4369071
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExecutor.java 7a01992
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcFromInitializer.java 1ecd152
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java 39e8edd
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/IncrementalImport.java PRE-CREATION
> connector/connector-generic-jdbc/src/main/resources/generic-jdbc-connector-config.properties 6a2159b
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestFromInitializer.java 52003ab
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestGenericJdbcConnector.java PRE-CREATION
> test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/IncrementalImportTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/31831/diff/
>
>
> Testing
> -------
>
> I've added several new unit tests and integration test for the newly added functionality.
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 31831: SQOOP-1805 GenericJdbcConnector:
FromJobConfiguration and Extractor changes if any to support delta read
Posted by Veena Basavaraj <vy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31831/#review76390
-----------------------------------------------------------
I have one generic suggestion on this RB.
We do not use IMORT , EXPORT anymore, can we make the config name more generic, Incremental fetch or read ?
- Veena Basavaraj
On March 7, 2015, 8:23 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31831/
> -----------------------------------------------------------
>
> (Updated March 7, 2015, 8:23 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1805
> https://issues.apache.org/jira/browse/SQOOP-1805
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I've added incremental options similar to Sqoop 1 and alter our partitioning function to take advantage of them. The method was relatively hard to change, so I've ended up rewriting it completely.
>
> The connector is not persisting the last values back to configuration object as the functionality is not exposed by Sqoop yet.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/sqoop/error/code/GenericJdbcConnectorError.java 03bc104
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java 4369071
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExecutor.java 7a01992
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcFromInitializer.java 1ecd152
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java 39e8edd
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/IncrementalImport.java PRE-CREATION
> connector/connector-generic-jdbc/src/main/resources/generic-jdbc-connector-config.properties 6a2159b
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestFromInitializer.java 52003ab
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestGenericJdbcConnector.java PRE-CREATION
> test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/IncrementalImportTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/31831/diff/
>
>
> Testing
> -------
>
> I've added several new unit tests and integration test for the newly added functionality.
>
>
> Thanks,
>
> Jarek Cecho
>
>