You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Jarek Cecho <ja...@apache.org> on 2013/02/05 20:47:04 UTC
Review Request: SQOOP-864 Sqoop2: Introduce ETL context objects
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9311/
-----------------------------------------------------------
Review request for Sqoop.
Description
-------
This patch seems to be quite enormous, but the logic is fairly simple. I've created family of Context classes - one parent ActorContext and than for each ETL separate child. My reasoning is that each ETL class needs different set of arguments and thus should have it's own Context class. Then I've went and fix all implementations of the ETL classes to conform with the changes.
Each context class is wrapping all parameters in a generic and extensible way. The only parameters that are not part of the Context objects are configuration objects. I wanted to move them into Context classes as well, but that did not work well with the generics, so I kept them separate.
This addresses bug SQOOP-864.
https://issues.apache.org/jira/browse/SQOOP-864
Diffs
-----
common/src/main/java/org/apache/sqoop/etl/io/DataReader.java PRE-CREATION
common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/ActorContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/InitializerContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java PRE-CREATION
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportDestroyer.java 2d53bdd0715de86d92ef5286855c8805301caadd
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java 06fbc519ab46ea01087e93398292a749b70630e2
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java 5f8e129d43f208a434527edfb771daf67bdf98af
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportDestroyer.java f7043ea2bd53409514ac7eb5197f65a076f683b9
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java 9db332865f1fcd7e21c48d3fad0c38d2a9f7fb8d
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java a509e2b6dc1b936f9988865cbe44610d77674ee6
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java b5dbc6168093f277de557051fee0afc6c2386c6c
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportInitializer.java c876780d076add6f6644bd2ab7ecb403676328c0
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java 5892cbbe03bd771f8f92cb21a722e4f6607e8107
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java 0be713e81de7ef522b55b1e8d63483cbb9db318e
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java f3c1d90efacb3ba5790e9b216cddb7a7e7bae65c
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java 77c47398ff3e046a2405765fb3abfb929b8704f4
core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 60c8f86636787369909a4549d7380f020d71f82b
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportPartitioner.java 71e00606d05fdaa91394f9da954321c2c58a02ab
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceExportExtractor.java 45b616678f95737a89ec5c46b589187752123a22
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java a5d6b9ca0510a2c1e0d334dacddbba8866bcc325
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextExportExtractor.java ed30c91c57f383224138276ddf7e85405d5e0240
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java 490b1c213ae4fa5e3860c49a551ba7a3c5372407
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java f4ef95ad8caf3e82f6860d134f87694bbcc20484
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java d191e038a863a64c180403d593d5b0835f3232fa
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java ce847f41b6d67aaa972816fa94dda3cbbd6a49ff
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java c5f3abdd560474a624d909754b3abd05c0e538a2
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java dbd1bbfaaeb0920202b2ba66d9b70ea316b4aaea
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java 6e1c9588975aa40aaee202f543d7f4a73be03c37
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 427132e20bb68b1c7e20f390f6edfa3277fbff3f
execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java 1a300aeb1d2a848993e3bce690aad906127ee14e
spi/src/main/java/org/apache/sqoop/job/etl/Destroyer.java cf2ed9acb1463f0dfbe8660d853c7d4ac80a8124
spi/src/main/java/org/apache/sqoop/job/etl/Extractor.java 300cf4eb4e89f7855344934feca66962c850d850
spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java 3fb6be03c2e3c989052dc23a9d831cc8d72af1da
spi/src/main/java/org/apache/sqoop/job/etl/Loader.java 024be9453c4202a17a8eaabead14c845b71975fd
spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java bfb4bf29fc5122f2aea11a6cb2059fb6f7eaa229
spi/src/main/java/org/apache/sqoop/job/io/DataReader.java a50f59108cf2af3520ca2119e8123a083eb4b94d
spi/src/main/java/org/apache/sqoop/job/io/DataWriter.java 30a0c7ceb925a2576746cbd879672104b9f3d3ba
Diff: https://reviews.apache.org/r/9311/diff/
Testing
-------
* mvn verify passes for both hadoop 1.x and 2.x
* I've verified the changes on real cluster
Thanks,
Jarek Cecho
Re: Review Request: SQOOP-864 Sqoop2: Introduce ETL context objects
Posted by Cheolsoo Park <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9311/#review16403
-----------------------------------------------------------
Ship it!
Ship It!
- Cheolsoo Park
On Feb. 10, 2013, 8:47 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9311/
> -----------------------------------------------------------
>
> (Updated Feb. 10, 2013, 8:47 p.m.)
>
>
> Review request for Sqoop.
>
>
> Description
> -------
>
> This patch seems to be quite enormous, but the logic is fairly simple. I've created family of Context classes - one parent ActorContext and than for each ETL separate child. My reasoning is that each ETL class needs different set of arguments and thus should have it's own Context class. Then I've went and fix all implementations of the ETL classes to conform with the changes.
>
> Each context class is wrapping all parameters in a generic and extensible way. The only parameters that are not part of the Context objects are configuration objects. I wanted to move them into Context classes as well, but that did not work well with the generics, so I kept them separate.
>
>
> This addresses bug SQOOP-864.
> https://issues.apache.org/jira/browse/SQOOP-864
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/sqoop/etl/io/DataReader.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/ActorContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/InitializerContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java PRE-CREATION
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportDestroyer.java 2d53bdd0715de86d92ef5286855c8805301caadd
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java 06fbc519ab46ea01087e93398292a749b70630e2
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java 5f8e129d43f208a434527edfb771daf67bdf98af
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportDestroyer.java f7043ea2bd53409514ac7eb5197f65a076f683b9
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java 9db332865f1fcd7e21c48d3fad0c38d2a9f7fb8d
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java a509e2b6dc1b936f9988865cbe44610d77674ee6
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java b5dbc6168093f277de557051fee0afc6c2386c6c
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportInitializer.java c876780d076add6f6644bd2ab7ecb403676328c0
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java 5892cbbe03bd771f8f92cb21a722e4f6607e8107
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java 0be713e81de7ef522b55b1e8d63483cbb9db318e
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java f3c1d90efacb3ba5790e9b216cddb7a7e7bae65c
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java 77c47398ff3e046a2405765fb3abfb929b8704f4
> core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 6aab2db49e55c3152149d61f29dcf8d8a3f95ba5
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportPartitioner.java 71e00606d05fdaa91394f9da954321c2c58a02ab
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceExportExtractor.java 45b616678f95737a89ec5c46b589187752123a22
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java a5d6b9ca0510a2c1e0d334dacddbba8866bcc325
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextExportExtractor.java ed30c91c57f383224138276ddf7e85405d5e0240
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java 490b1c213ae4fa5e3860c49a551ba7a3c5372407
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java f4ef95ad8caf3e82f6860d134f87694bbcc20484
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java d191e038a863a64c180403d593d5b0835f3232fa
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java ce847f41b6d67aaa972816fa94dda3cbbd6a49ff
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java c5f3abdd560474a624d909754b3abd05c0e538a2
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java dbd1bbfaaeb0920202b2ba66d9b70ea316b4aaea
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java 6e1c9588975aa40aaee202f543d7f4a73be03c37
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 427132e20bb68b1c7e20f390f6edfa3277fbff3f
> execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java 1a300aeb1d2a848993e3bce690aad906127ee14e
> spi/src/main/java/org/apache/sqoop/job/etl/Destroyer.java cf2ed9acb1463f0dfbe8660d853c7d4ac80a8124
> spi/src/main/java/org/apache/sqoop/job/etl/Extractor.java 300cf4eb4e89f7855344934feca66962c850d850
> spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java 3fb6be03c2e3c989052dc23a9d831cc8d72af1da
> spi/src/main/java/org/apache/sqoop/job/etl/Loader.java 024be9453c4202a17a8eaabead14c845b71975fd
> spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java bfb4bf29fc5122f2aea11a6cb2059fb6f7eaa229
> spi/src/main/java/org/apache/sqoop/job/io/DataReader.java a50f59108cf2af3520ca2119e8123a083eb4b94d
> spi/src/main/java/org/apache/sqoop/job/io/DataWriter.java 30a0c7ceb925a2576746cbd879672104b9f3d3ba
>
> Diff: https://reviews.apache.org/r/9311/diff/
>
>
> Testing
> -------
>
> * mvn verify passes for both hadoop 1.x and 2.x
> * I've verified the changes on real cluster
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request: SQOOP-864 Sqoop2: Introduce ETL context objects
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9311/
-----------------------------------------------------------
(Updated Feb. 10, 2013, 8:47 p.m.)
Review request for Sqoop.
Changes
-------
Fixing missing one change.
Description
-------
This patch seems to be quite enormous, but the logic is fairly simple. I've created family of Context classes - one parent ActorContext and than for each ETL separate child. My reasoning is that each ETL class needs different set of arguments and thus should have it's own Context class. Then I've went and fix all implementations of the ETL classes to conform with the changes.
Each context class is wrapping all parameters in a generic and extensible way. The only parameters that are not part of the Context objects are configuration objects. I wanted to move them into Context classes as well, but that did not work well with the generics, so I kept them separate.
This addresses bug SQOOP-864.
https://issues.apache.org/jira/browse/SQOOP-864
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/etl/io/DataReader.java PRE-CREATION
common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/ActorContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/InitializerContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java PRE-CREATION
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportDestroyer.java 2d53bdd0715de86d92ef5286855c8805301caadd
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java 06fbc519ab46ea01087e93398292a749b70630e2
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java 5f8e129d43f208a434527edfb771daf67bdf98af
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportDestroyer.java f7043ea2bd53409514ac7eb5197f65a076f683b9
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java 9db332865f1fcd7e21c48d3fad0c38d2a9f7fb8d
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java a509e2b6dc1b936f9988865cbe44610d77674ee6
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java b5dbc6168093f277de557051fee0afc6c2386c6c
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportInitializer.java c876780d076add6f6644bd2ab7ecb403676328c0
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java 5892cbbe03bd771f8f92cb21a722e4f6607e8107
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java 0be713e81de7ef522b55b1e8d63483cbb9db318e
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java f3c1d90efacb3ba5790e9b216cddb7a7e7bae65c
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java 77c47398ff3e046a2405765fb3abfb929b8704f4
core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 6aab2db49e55c3152149d61f29dcf8d8a3f95ba5
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportPartitioner.java 71e00606d05fdaa91394f9da954321c2c58a02ab
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceExportExtractor.java 45b616678f95737a89ec5c46b589187752123a22
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java a5d6b9ca0510a2c1e0d334dacddbba8866bcc325
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextExportExtractor.java ed30c91c57f383224138276ddf7e85405d5e0240
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java 490b1c213ae4fa5e3860c49a551ba7a3c5372407
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java f4ef95ad8caf3e82f6860d134f87694bbcc20484
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java d191e038a863a64c180403d593d5b0835f3232fa
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java ce847f41b6d67aaa972816fa94dda3cbbd6a49ff
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java c5f3abdd560474a624d909754b3abd05c0e538a2
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java dbd1bbfaaeb0920202b2ba66d9b70ea316b4aaea
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java 6e1c9588975aa40aaee202f543d7f4a73be03c37
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 427132e20bb68b1c7e20f390f6edfa3277fbff3f
execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java 1a300aeb1d2a848993e3bce690aad906127ee14e
spi/src/main/java/org/apache/sqoop/job/etl/Destroyer.java cf2ed9acb1463f0dfbe8660d853c7d4ac80a8124
spi/src/main/java/org/apache/sqoop/job/etl/Extractor.java 300cf4eb4e89f7855344934feca66962c850d850
spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java 3fb6be03c2e3c989052dc23a9d831cc8d72af1da
spi/src/main/java/org/apache/sqoop/job/etl/Loader.java 024be9453c4202a17a8eaabead14c845b71975fd
spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java bfb4bf29fc5122f2aea11a6cb2059fb6f7eaa229
spi/src/main/java/org/apache/sqoop/job/io/DataReader.java a50f59108cf2af3520ca2119e8123a083eb4b94d
spi/src/main/java/org/apache/sqoop/job/io/DataWriter.java 30a0c7ceb925a2576746cbd879672104b9f3d3ba
Diff: https://reviews.apache.org/r/9311/diff/
Testing
-------
* mvn verify passes for both hadoop 1.x and 2.x
* I've verified the changes on real cluster
Thanks,
Jarek Cecho
Re: Review Request: SQOOP-864 Sqoop2: Introduce ETL context objects
Posted by Cheolsoo Park <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9311/#review16400
-----------------------------------------------------------
Jarcec, I think you missed one line. Do you mind fixing it?
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java
<https://reviews.apache.org/r/9311/#comment34887>
I guess you omit to change this?
- Cheolsoo Park
On Feb. 8, 2013, 9:29 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9311/
> -----------------------------------------------------------
>
> (Updated Feb. 8, 2013, 9:29 p.m.)
>
>
> Review request for Sqoop.
>
>
> Description
> -------
>
> This patch seems to be quite enormous, but the logic is fairly simple. I've created family of Context classes - one parent ActorContext and than for each ETL separate child. My reasoning is that each ETL class needs different set of arguments and thus should have it's own Context class. Then I've went and fix all implementations of the ETL classes to conform with the changes.
>
> Each context class is wrapping all parameters in a generic and extensible way. The only parameters that are not part of the Context objects are configuration objects. I wanted to move them into Context classes as well, but that did not work well with the generics, so I kept them separate.
>
>
> This addresses bug SQOOP-864.
> https://issues.apache.org/jira/browse/SQOOP-864
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/sqoop/etl/io/DataReader.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/ActorContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/InitializerContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java PRE-CREATION
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportDestroyer.java 2d53bdd0715de86d92ef5286855c8805301caadd
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java 06fbc519ab46ea01087e93398292a749b70630e2
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java 5f8e129d43f208a434527edfb771daf67bdf98af
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportDestroyer.java f7043ea2bd53409514ac7eb5197f65a076f683b9
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java 9db332865f1fcd7e21c48d3fad0c38d2a9f7fb8d
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java a509e2b6dc1b936f9988865cbe44610d77674ee6
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java b5dbc6168093f277de557051fee0afc6c2386c6c
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportInitializer.java c876780d076add6f6644bd2ab7ecb403676328c0
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java 5892cbbe03bd771f8f92cb21a722e4f6607e8107
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java 0be713e81de7ef522b55b1e8d63483cbb9db318e
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java f3c1d90efacb3ba5790e9b216cddb7a7e7bae65c
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java 77c47398ff3e046a2405765fb3abfb929b8704f4
> core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 6aab2db49e55c3152149d61f29dcf8d8a3f95ba5
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportPartitioner.java 71e00606d05fdaa91394f9da954321c2c58a02ab
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceExportExtractor.java 45b616678f95737a89ec5c46b589187752123a22
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java a5d6b9ca0510a2c1e0d334dacddbba8866bcc325
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextExportExtractor.java ed30c91c57f383224138276ddf7e85405d5e0240
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java 490b1c213ae4fa5e3860c49a551ba7a3c5372407
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java f4ef95ad8caf3e82f6860d134f87694bbcc20484
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java d191e038a863a64c180403d593d5b0835f3232fa
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java ce847f41b6d67aaa972816fa94dda3cbbd6a49ff
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java c5f3abdd560474a624d909754b3abd05c0e538a2
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java dbd1bbfaaeb0920202b2ba66d9b70ea316b4aaea
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java 6e1c9588975aa40aaee202f543d7f4a73be03c37
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 427132e20bb68b1c7e20f390f6edfa3277fbff3f
> execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java 1a300aeb1d2a848993e3bce690aad906127ee14e
> spi/src/main/java/org/apache/sqoop/job/etl/Destroyer.java cf2ed9acb1463f0dfbe8660d853c7d4ac80a8124
> spi/src/main/java/org/apache/sqoop/job/etl/Extractor.java 300cf4eb4e89f7855344934feca66962c850d850
> spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java 3fb6be03c2e3c989052dc23a9d831cc8d72af1da
> spi/src/main/java/org/apache/sqoop/job/etl/Loader.java 024be9453c4202a17a8eaabead14c845b71975fd
> spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java bfb4bf29fc5122f2aea11a6cb2059fb6f7eaa229
> spi/src/main/java/org/apache/sqoop/job/io/DataReader.java a50f59108cf2af3520ca2119e8123a083eb4b94d
> spi/src/main/java/org/apache/sqoop/job/io/DataWriter.java 30a0c7ceb925a2576746cbd879672104b9f3d3ba
>
> Diff: https://reviews.apache.org/r/9311/diff/
>
>
> Testing
> -------
>
> * mvn verify passes for both hadoop 1.x and 2.x
> * I've verified the changes on real cluster
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request: SQOOP-864 Sqoop2: Introduce ETL context objects
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9311/
-----------------------------------------------------------
(Updated Feb. 8, 2013, 9:29 p.m.)
Review request for Sqoop.
Changes
-------
I've incorporated Cheolsoo's suggestions.
Description
-------
This patch seems to be quite enormous, but the logic is fairly simple. I've created family of Context classes - one parent ActorContext and than for each ETL separate child. My reasoning is that each ETL class needs different set of arguments and thus should have it's own Context class. Then I've went and fix all implementations of the ETL classes to conform with the changes.
Each context class is wrapping all parameters in a generic and extensible way. The only parameters that are not part of the Context objects are configuration objects. I wanted to move them into Context classes as well, but that did not work well with the generics, so I kept them separate.
This addresses bug SQOOP-864.
https://issues.apache.org/jira/browse/SQOOP-864
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/etl/io/DataReader.java PRE-CREATION
common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/ActorContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/InitializerContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java PRE-CREATION
common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java PRE-CREATION
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportDestroyer.java 2d53bdd0715de86d92ef5286855c8805301caadd
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java 06fbc519ab46ea01087e93398292a749b70630e2
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java 5f8e129d43f208a434527edfb771daf67bdf98af
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportDestroyer.java f7043ea2bd53409514ac7eb5197f65a076f683b9
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java 9db332865f1fcd7e21c48d3fad0c38d2a9f7fb8d
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java a509e2b6dc1b936f9988865cbe44610d77674ee6
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java b5dbc6168093f277de557051fee0afc6c2386c6c
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportInitializer.java c876780d076add6f6644bd2ab7ecb403676328c0
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java 5892cbbe03bd771f8f92cb21a722e4f6607e8107
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java 0be713e81de7ef522b55b1e8d63483cbb9db318e
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java f3c1d90efacb3ba5790e9b216cddb7a7e7bae65c
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java 77c47398ff3e046a2405765fb3abfb929b8704f4
core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 6aab2db49e55c3152149d61f29dcf8d8a3f95ba5
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportPartitioner.java 71e00606d05fdaa91394f9da954321c2c58a02ab
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceExportExtractor.java 45b616678f95737a89ec5c46b589187752123a22
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java a5d6b9ca0510a2c1e0d334dacddbba8866bcc325
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextExportExtractor.java ed30c91c57f383224138276ddf7e85405d5e0240
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java 490b1c213ae4fa5e3860c49a551ba7a3c5372407
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java f4ef95ad8caf3e82f6860d134f87694bbcc20484
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java d191e038a863a64c180403d593d5b0835f3232fa
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java ce847f41b6d67aaa972816fa94dda3cbbd6a49ff
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java c5f3abdd560474a624d909754b3abd05c0e538a2
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java dbd1bbfaaeb0920202b2ba66d9b70ea316b4aaea
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java 6e1c9588975aa40aaee202f543d7f4a73be03c37
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 427132e20bb68b1c7e20f390f6edfa3277fbff3f
execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java 1a300aeb1d2a848993e3bce690aad906127ee14e
spi/src/main/java/org/apache/sqoop/job/etl/Destroyer.java cf2ed9acb1463f0dfbe8660d853c7d4ac80a8124
spi/src/main/java/org/apache/sqoop/job/etl/Extractor.java 300cf4eb4e89f7855344934feca66962c850d850
spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java 3fb6be03c2e3c989052dc23a9d831cc8d72af1da
spi/src/main/java/org/apache/sqoop/job/etl/Loader.java 024be9453c4202a17a8eaabead14c845b71975fd
spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java bfb4bf29fc5122f2aea11a6cb2059fb6f7eaa229
spi/src/main/java/org/apache/sqoop/job/io/DataReader.java a50f59108cf2af3520ca2119e8123a083eb4b94d
spi/src/main/java/org/apache/sqoop/job/io/DataWriter.java 30a0c7ceb925a2576746cbd879672104b9f3d3ba
Diff: https://reviews.apache.org/r/9311/diff/
Testing
-------
* mvn verify passes for both hadoop 1.x and 2.x
* I've verified the changes on real cluster
Thanks,
Jarek Cecho
Re: Review Request: SQOOP-864 Sqoop2: Introduce ETL context objects
Posted by Cheolsoo Park <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9311/#review16206
-----------------------------------------------------------
Jarcec, I fully support this. I agree that this will help to avoid breaking the compatibility of public API in the future.
I just have one suggestion, which can be applied to several places. Please see my comments inline.
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java
<https://reviews.apache.org/r/9311/#comment34638>
Can you add the "getString(...)" method to ActorContext so that we can shorten this to "context.getString(..)"?
In ActorContext.getString(...), you can return "context.getContext().getString(...)".
I think this is cleaner.
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java
<https://reviews.apache.org/r/9311/#comment34639>
The same as above.
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java
<https://reviews.apache.org/r/9311/#comment34640>
The same as above.
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java
<https://reviews.apache.org/r/9311/#comment34641>
The same as above.
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java
<https://reviews.apache.org/r/9311/#comment34642>
The same as above.
- Cheolsoo Park
On Feb. 5, 2013, 7:46 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9311/
> -----------------------------------------------------------
>
> (Updated Feb. 5, 2013, 7:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Description
> -------
>
> This patch seems to be quite enormous, but the logic is fairly simple. I've created family of Context classes - one parent ActorContext and than for each ETL separate child. My reasoning is that each ETL class needs different set of arguments and thus should have it's own Context class. Then I've went and fix all implementations of the ETL classes to conform with the changes.
>
> Each context class is wrapping all parameters in a generic and extensible way. The only parameters that are not part of the Context objects are configuration objects. I wanted to move them into Context classes as well, but that did not work well with the generics, so I kept them separate.
>
>
> This addresses bug SQOOP-864.
> https://issues.apache.org/jira/browse/SQOOP-864
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/sqoop/etl/io/DataReader.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/ActorContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/InitializerContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java PRE-CREATION
> common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java PRE-CREATION
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportDestroyer.java 2d53bdd0715de86d92ef5286855c8805301caadd
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java 06fbc519ab46ea01087e93398292a749b70630e2
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java 5f8e129d43f208a434527edfb771daf67bdf98af
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportDestroyer.java f7043ea2bd53409514ac7eb5197f65a076f683b9
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java 9db332865f1fcd7e21c48d3fad0c38d2a9f7fb8d
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java a509e2b6dc1b936f9988865cbe44610d77674ee6
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java b5dbc6168093f277de557051fee0afc6c2386c6c
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportInitializer.java c876780d076add6f6644bd2ab7ecb403676328c0
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java 5892cbbe03bd771f8f92cb21a722e4f6607e8107
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java 0be713e81de7ef522b55b1e8d63483cbb9db318e
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java f3c1d90efacb3ba5790e9b216cddb7a7e7bae65c
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java 77c47398ff3e046a2405765fb3abfb929b8704f4
> core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 60c8f86636787369909a4549d7380f020d71f82b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportPartitioner.java 71e00606d05fdaa91394f9da954321c2c58a02ab
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceExportExtractor.java 45b616678f95737a89ec5c46b589187752123a22
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java a5d6b9ca0510a2c1e0d334dacddbba8866bcc325
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextExportExtractor.java ed30c91c57f383224138276ddf7e85405d5e0240
> execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java 490b1c213ae4fa5e3860c49a551ba7a3c5372407
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java f4ef95ad8caf3e82f6860d134f87694bbcc20484
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java d191e038a863a64c180403d593d5b0835f3232fa
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java ce847f41b6d67aaa972816fa94dda3cbbd6a49ff
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java c5f3abdd560474a624d909754b3abd05c0e538a2
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java dbd1bbfaaeb0920202b2ba66d9b70ea316b4aaea
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java 6e1c9588975aa40aaee202f543d7f4a73be03c37
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 427132e20bb68b1c7e20f390f6edfa3277fbff3f
> execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java 1a300aeb1d2a848993e3bce690aad906127ee14e
> spi/src/main/java/org/apache/sqoop/job/etl/Destroyer.java cf2ed9acb1463f0dfbe8660d853c7d4ac80a8124
> spi/src/main/java/org/apache/sqoop/job/etl/Extractor.java 300cf4eb4e89f7855344934feca66962c850d850
> spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java 3fb6be03c2e3c989052dc23a9d831cc8d72af1da
> spi/src/main/java/org/apache/sqoop/job/etl/Loader.java 024be9453c4202a17a8eaabead14c845b71975fd
> spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java bfb4bf29fc5122f2aea11a6cb2059fb6f7eaa229
> spi/src/main/java/org/apache/sqoop/job/io/DataReader.java a50f59108cf2af3520ca2119e8123a083eb4b94d
> spi/src/main/java/org/apache/sqoop/job/io/DataWriter.java 30a0c7ceb925a2576746cbd879672104b9f3d3ba
>
> Diff: https://reviews.apache.org/r/9311/diff/
>
>
> Testing
> -------
>
> * mvn verify passes for both hadoop 1.x and 2.x
> * I've verified the changes on real cluster
>
>
> Thanks,
>
> Jarek Cecho
>
>