You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Jarek Cecho <ja...@apache.org> on 2013/07/04 00:46:10 UTC
Review Request 12261: SQOOP-1121 Sqoop2: Serialize schema and make them
available in the MR job.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12261/
-----------------------------------------------------------
Review request for Sqoop.
Bugs: SQOOP-1121
https://issues.apache.org/jira/browse/SQOOP-1121
Repository: sqoop-sqoop2
Description
-------
The patch seems big, but it's just touching a lot of files. Couple of high level notes:
* I've added serialization of the schema into mapreduce job
* I've refactored a bit ConfigurationUtils by providing better encapsulation and adding unit tests for that class.
* I've added the schema to all MR context objects (that are passed to extractor, loader, ...)
* Currently only Connector schema is being used everywhere, I'm expecting to change that once Framework (=hio) schema will be in place.
Diffs
-----
common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java 10cfb103b97a367c87c9ab264bf83035a5547660
common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java f9d7a8b57b999db4531e46ce62e2ac535e580e3f
common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java dad19f15672375084d3c9d6736c19628853222c9
common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java 5e7cea75cb5d683c4bdc47674a7b6ecc56881b32
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java 50a32d91f3a6f2290a6519754de9427e167df20b
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java 54ffe5ba44a12ca71b08cd93a9ed347255cb8d25
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java d46e4dde8b79167707694e1b88117caf71053833
core/src/main/java/org/apache/sqoop/framework/JobManager.java 58d6c104bbc589f33857f43f8831448635758ce8
execution/mapreduce/pom.xml 31da5f11887415db85e26c6ccb2500383d9b2635
execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java e2b3ce84484c8a5a984a8b31a4d83e5ca03c0ba4
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java 64ec4375fd4c2edb48f605b4997cccd29d420a22
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java 4493a45f720c015d285030e5b7b1382e1f3f16e7
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0721b7e20f00c0d8559356919ca79bbdcbe669b5
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7715d5f7be23c838d337f2c117af330843e99e8f
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java d47f861035a9eb199d166dc9f54542654b63a9b8
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java 58c306824ca307eb062e6664cb1962072cb5526b
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java b9b063193d39b005a1c36bf7a5399559c39ac925
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java ee03427ba0ca7009c0ea2edf15e4f482f6584ed2
execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java PRE-CREATION
execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java d2b501eb56ae0708754047150e0618e1f6470395
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java 0e8c9f72339f1f6d82ce2a7c49cc1645369f480d
Diff: https://reviews.apache.org/r/12261/diff/
Testing
-------
Provided new unit test for the util class ConfigurationUtils. Both unit and integration tests seems to be working, plus I've tested the changes on real cluster.
Thanks,
Jarek Cecho
Re: Review Request 12261: SQOOP-1121 Sqoop2: Serialize schema and make them
available in the MR job.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12261/#review22958
-----------------------------------------------------------
Ship it!
Ship It!
- Hari Shreedharan
On July 10, 2013, 3:05 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12261/
> -----------------------------------------------------------
>
> (Updated July 10, 2013, 3:05 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1121
> https://issues.apache.org/jira/browse/SQOOP-1121
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> The patch seems big, but it's just touching a lot of files. Couple of high level notes:
>
> * I've added serialization of the schema into mapreduce job
> * I've refactored a bit ConfigurationUtils by providing better encapsulation and adding unit tests for that class.
> * I've added the schema to all MR context objects (that are passed to extractor, loader, ...)
> * Currently only Connector schema is being used everywhere, I'm expecting to change that once Framework (=hio) schema will be in place.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java 10cfb103b97a367c87c9ab264bf83035a5547660
> common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java f9d7a8b57b999db4531e46ce62e2ac535e580e3f
> common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java dad19f15672375084d3c9d6736c19628853222c9
> common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java 5e7cea75cb5d683c4bdc47674a7b6ecc56881b32
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java 50a32d91f3a6f2290a6519754de9427e167df20b
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java 54ffe5ba44a12ca71b08cd93a9ed347255cb8d25
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java 0afec49f1b46d2b0c8b30a029507f5de6f7e7ee6
> core/src/main/java/org/apache/sqoop/framework/JobManager.java 58d6c104bbc589f33857f43f8831448635758ce8
> execution/mapreduce/pom.xml 31da5f11887415db85e26c6ccb2500383d9b2635
> execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java e2b3ce84484c8a5a984a8b31a4d83e5ca03c0ba4
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java 64ec4375fd4c2edb48f605b4997cccd29d420a22
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java 4493a45f720c015d285030e5b7b1382e1f3f16e7
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0721b7e20f00c0d8559356919ca79bbdcbe669b5
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7715d5f7be23c838d337f2c117af330843e99e8f
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java d47f861035a9eb199d166dc9f54542654b63a9b8
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java 58c306824ca307eb062e6664cb1962072cb5526b
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java b9b063193d39b005a1c36bf7a5399559c39ac925
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java ee03427ba0ca7009c0ea2edf15e4f482f6584ed2
> execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java PRE-CREATION
> execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java d2b501eb56ae0708754047150e0618e1f6470395
> submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java 0e8c9f72339f1f6d82ce2a7c49cc1645369f480d
>
> Diff: https://reviews.apache.org/r/12261/diff/
>
>
> Testing
> -------
>
> Provided new unit test for the util class ConfigurationUtils. Both unit and integration tests seems to be working, plus I've tested the changes on real cluster.
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 12261: SQOOP-1121 Sqoop2: Serialize schema and make them
available in the MR job.
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12261/
-----------------------------------------------------------
(Updated July 10, 2013, 3:05 p.m.)
Review request for Sqoop.
Changes
-------
Propagating schema to Partitioner.
Bugs: SQOOP-1121
https://issues.apache.org/jira/browse/SQOOP-1121
Repository: sqoop-sqoop2
Description
-------
The patch seems big, but it's just touching a lot of files. Couple of high level notes:
* I've added serialization of the schema into mapreduce job
* I've refactored a bit ConfigurationUtils by providing better encapsulation and adding unit tests for that class.
* I've added the schema to all MR context objects (that are passed to extractor, loader, ...)
* Currently only Connector schema is being used everywhere, I'm expecting to change that once Framework (=hio) schema will be in place.
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java 10cfb103b97a367c87c9ab264bf83035a5547660
common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java f9d7a8b57b999db4531e46ce62e2ac535e580e3f
common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java dad19f15672375084d3c9d6736c19628853222c9
common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java 5e7cea75cb5d683c4bdc47674a7b6ecc56881b32
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java 50a32d91f3a6f2290a6519754de9427e167df20b
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java 54ffe5ba44a12ca71b08cd93a9ed347255cb8d25
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java 0afec49f1b46d2b0c8b30a029507f5de6f7e7ee6
core/src/main/java/org/apache/sqoop/framework/JobManager.java 58d6c104bbc589f33857f43f8831448635758ce8
execution/mapreduce/pom.xml 31da5f11887415db85e26c6ccb2500383d9b2635
execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java e2b3ce84484c8a5a984a8b31a4d83e5ca03c0ba4
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java 64ec4375fd4c2edb48f605b4997cccd29d420a22
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java 4493a45f720c015d285030e5b7b1382e1f3f16e7
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0721b7e20f00c0d8559356919ca79bbdcbe669b5
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7715d5f7be23c838d337f2c117af330843e99e8f
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java d47f861035a9eb199d166dc9f54542654b63a9b8
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java 58c306824ca307eb062e6664cb1962072cb5526b
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java b9b063193d39b005a1c36bf7a5399559c39ac925
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java ee03427ba0ca7009c0ea2edf15e4f482f6584ed2
execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java PRE-CREATION
execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java d2b501eb56ae0708754047150e0618e1f6470395
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java 0e8c9f72339f1f6d82ce2a7c49cc1645369f480d
Diff: https://reviews.apache.org/r/12261/diff/
Testing
-------
Provided new unit test for the util class ConfigurationUtils. Both unit and integration tests seems to be working, plus I've tested the changes on real cluster.
Thanks,
Jarek Cecho
Re: Review Request 12261: SQOOP-1121 Sqoop2: Serialize schema and make them
available in the MR job.
Posted by Jarek Cecho <ja...@apache.org>.
> On July 10, 2013, 6:21 a.m., Hari Shreedharan wrote:
> > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java, line 69
> > <https://reviews.apache.org/r/12261/diff/2/?file=319204#file319204line69>
> >
> > Shouldn't the schema be passed to the partitioner too? Maybe useful for some connectors to create partitions.
That should definitely be the case, thank you for pointing that out Hari!
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12261/#review22948
-----------------------------------------------------------
On July 10, 2013, 3:05 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12261/
> -----------------------------------------------------------
>
> (Updated July 10, 2013, 3:05 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1121
> https://issues.apache.org/jira/browse/SQOOP-1121
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> The patch seems big, but it's just touching a lot of files. Couple of high level notes:
>
> * I've added serialization of the schema into mapreduce job
> * I've refactored a bit ConfigurationUtils by providing better encapsulation and adding unit tests for that class.
> * I've added the schema to all MR context objects (that are passed to extractor, loader, ...)
> * Currently only Connector schema is being used everywhere, I'm expecting to change that once Framework (=hio) schema will be in place.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java 10cfb103b97a367c87c9ab264bf83035a5547660
> common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java f9d7a8b57b999db4531e46ce62e2ac535e580e3f
> common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java dad19f15672375084d3c9d6736c19628853222c9
> common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java 5e7cea75cb5d683c4bdc47674a7b6ecc56881b32
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java 50a32d91f3a6f2290a6519754de9427e167df20b
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java 54ffe5ba44a12ca71b08cd93a9ed347255cb8d25
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java 0afec49f1b46d2b0c8b30a029507f5de6f7e7ee6
> core/src/main/java/org/apache/sqoop/framework/JobManager.java 58d6c104bbc589f33857f43f8831448635758ce8
> execution/mapreduce/pom.xml 31da5f11887415db85e26c6ccb2500383d9b2635
> execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java e2b3ce84484c8a5a984a8b31a4d83e5ca03c0ba4
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java 64ec4375fd4c2edb48f605b4997cccd29d420a22
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java 4493a45f720c015d285030e5b7b1382e1f3f16e7
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0721b7e20f00c0d8559356919ca79bbdcbe669b5
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7715d5f7be23c838d337f2c117af330843e99e8f
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java d47f861035a9eb199d166dc9f54542654b63a9b8
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java 58c306824ca307eb062e6664cb1962072cb5526b
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java b9b063193d39b005a1c36bf7a5399559c39ac925
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java ee03427ba0ca7009c0ea2edf15e4f482f6584ed2
> execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java PRE-CREATION
> execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java d2b501eb56ae0708754047150e0618e1f6470395
> submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java 0e8c9f72339f1f6d82ce2a7c49cc1645369f480d
>
> Diff: https://reviews.apache.org/r/12261/diff/
>
>
> Testing
> -------
>
> Provided new unit test for the util class ConfigurationUtils. Both unit and integration tests seems to be working, plus I've tested the changes on real cluster.
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 12261: SQOOP-1121 Sqoop2: Serialize schema and make them
available in the MR job.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12261/#review22948
-----------------------------------------------------------
Jarcec,
Looks good. Just 1 comment below. Once you fix that, please attach the patch to the jira.
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java
<https://reviews.apache.org/r/12261/#comment46733>
Shouldn't the schema be passed to the partitioner too? Maybe useful for some connectors to create partitions.
- Hari Shreedharan
On July 9, 2013, 4:18 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12261/
> -----------------------------------------------------------
>
> (Updated July 9, 2013, 4:18 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1121
> https://issues.apache.org/jira/browse/SQOOP-1121
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> The patch seems big, but it's just touching a lot of files. Couple of high level notes:
>
> * I've added serialization of the schema into mapreduce job
> * I've refactored a bit ConfigurationUtils by providing better encapsulation and adding unit tests for that class.
> * I've added the schema to all MR context objects (that are passed to extractor, loader, ...)
> * Currently only Connector schema is being used everywhere, I'm expecting to change that once Framework (=hio) schema will be in place.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java 10cfb103b97a367c87c9ab264bf83035a5547660
> common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java f9d7a8b57b999db4531e46ce62e2ac535e580e3f
> common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java dad19f15672375084d3c9d6736c19628853222c9
> common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java 5e7cea75cb5d683c4bdc47674a7b6ecc56881b32
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java 50a32d91f3a6f2290a6519754de9427e167df20b
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java 54ffe5ba44a12ca71b08cd93a9ed347255cb8d25
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java 0afec49f1b46d2b0c8b30a029507f5de6f7e7ee6
> core/src/main/java/org/apache/sqoop/framework/JobManager.java 58d6c104bbc589f33857f43f8831448635758ce8
> execution/mapreduce/pom.xml 31da5f11887415db85e26c6ccb2500383d9b2635
> execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java e2b3ce84484c8a5a984a8b31a4d83e5ca03c0ba4
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java 64ec4375fd4c2edb48f605b4997cccd29d420a22
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java 4493a45f720c015d285030e5b7b1382e1f3f16e7
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0721b7e20f00c0d8559356919ca79bbdcbe669b5
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7715d5f7be23c838d337f2c117af330843e99e8f
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java d47f861035a9eb199d166dc9f54542654b63a9b8
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java 58c306824ca307eb062e6664cb1962072cb5526b
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java b9b063193d39b005a1c36bf7a5399559c39ac925
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java ee03427ba0ca7009c0ea2edf15e4f482f6584ed2
> execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java PRE-CREATION
> execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java d2b501eb56ae0708754047150e0618e1f6470395
> submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java 0e8c9f72339f1f6d82ce2a7c49cc1645369f480d
>
> Diff: https://reviews.apache.org/r/12261/diff/
>
>
> Testing
> -------
>
> Provided new unit test for the util class ConfigurationUtils. Both unit and integration tests seems to be working, plus I've tested the changes on real cluster.
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request 12261: SQOOP-1121 Sqoop2: Serialize schema and make them
available in the MR job.
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12261/
-----------------------------------------------------------
(Updated July 9, 2013, 4:18 p.m.)
Review request for Sqoop.
Changes
-------
Rebasing on current head of the sqoop2 branch.
Bugs: SQOOP-1121
https://issues.apache.org/jira/browse/SQOOP-1121
Repository: sqoop-sqoop2
Description
-------
The patch seems big, but it's just touching a lot of files. Couple of high level notes:
* I've added serialization of the schema into mapreduce job
* I've refactored a bit ConfigurationUtils by providing better encapsulation and adding unit tests for that class.
* I've added the schema to all MR context objects (that are passed to extractor, loader, ...)
* Currently only Connector schema is being used everywhere, I'm expecting to change that once Framework (=hio) schema will be in place.
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java 10cfb103b97a367c87c9ab264bf83035a5547660
common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java f9d7a8b57b999db4531e46ce62e2ac535e580e3f
common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java dad19f15672375084d3c9d6736c19628853222c9
common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java 5e7cea75cb5d683c4bdc47674a7b6ecc56881b32
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java 50a32d91f3a6f2290a6519754de9427e167df20b
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java 54ffe5ba44a12ca71b08cd93a9ed347255cb8d25
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java 0afec49f1b46d2b0c8b30a029507f5de6f7e7ee6
core/src/main/java/org/apache/sqoop/framework/JobManager.java 58d6c104bbc589f33857f43f8831448635758ce8
execution/mapreduce/pom.xml 31da5f11887415db85e26c6ccb2500383d9b2635
execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java e2b3ce84484c8a5a984a8b31a4d83e5ca03c0ba4
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java 64ec4375fd4c2edb48f605b4997cccd29d420a22
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java 4493a45f720c015d285030e5b7b1382e1f3f16e7
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0721b7e20f00c0d8559356919ca79bbdcbe669b5
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7715d5f7be23c838d337f2c117af330843e99e8f
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java d47f861035a9eb199d166dc9f54542654b63a9b8
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java 58c306824ca307eb062e6664cb1962072cb5526b
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java b9b063193d39b005a1c36bf7a5399559c39ac925
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java ee03427ba0ca7009c0ea2edf15e4f482f6584ed2
execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java PRE-CREATION
execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java d2b501eb56ae0708754047150e0618e1f6470395
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java 0e8c9f72339f1f6d82ce2a7c49cc1645369f480d
Diff: https://reviews.apache.org/r/12261/diff/
Testing
-------
Provided new unit test for the util class ConfigurationUtils. Both unit and integration tests seems to be working, plus I've tested the changes on real cluster.
Thanks,
Jarek Cecho