You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Jarek Cecho <ja...@apache.org> on 2012/12/04 18:26:24 UTC
Review Request: SQOOP-737 Give user option to set number of extractors and
loaders
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8339/
-----------------------------------------------------------
Review request for Sqoop.
Description
-------
I've implemented end-to-end behaviour to cover setting number of extractors and loaders. Current default is to have 10 extractors (=mappers) and equal number of loaders as extractors (= no reduce phase). Setting any number of loaders will add reduce phase.
This addresses bug SQOOP-737.
https://issues.apache.org/jira/browse/SQOOP-737
Diffs
-----
common/src/main/java/org/apache/sqoop/common/MutableContext.java 238bbfd4d621bd0b091a5a1850d7926da6431524
common/src/main/java/org/apache/sqoop/common/MutableMapContext.java cd9d3e3f9a6a79cb27bf21fd605331727d60b09e
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java a6d3b52fed2fd7a3f34e787603a3feeb1a0c6fd8
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java d5db190f831d977f4414fb0e27498885dbd03636
core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 0cd696982772e9dd80e7584435cd5be0106bfd97
core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java fb6b6a94722baf5c29292d72ed8908cc1e2140d7
core/src/main/java/org/apache/sqoop/framework/configuration/ImportJobConfiguration.java 8c4dcf168a8df70b727d58e108d3df987e643bf3
core/src/main/java/org/apache/sqoop/framework/configuration/ThrottlingForm.java PRE-CREATION
core/src/main/resources/framework-resources.properties 5ddf32debcdcc56d332f5316b4829eaf5903b379
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 4a5b305bbb628e47860e98c3dda8cbb28c1416e9
execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java d899fcef662ed69b567762e55c40840c66fc2387
execution/mapreduce/src/main/java/org/apache/sqoop/job/MapreduceExecutionError.java aa1f7332079c919d162bcf81ad01f35b7ca00386
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 8fcdc99daa08a02775ed2ba0708eccfdfdbe078b
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java 21a2be9d7ca4fa5d3ff9dc95ad7a50c622560691
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 745a3a42bff9d74a2d62438ff0cbcc8babb00e26
spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java 3a525c4fed5fdc8372f83b72c5a37538955618b8
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java a64a477577b84afb0d0848516186682dda9b6310
Diff: https://reviews.apache.org/r/8339/diff/
Testing
-------
Unit tests are passing and I've tested it on real cluster.
Thanks,
Jarek Cecho
Re: Review Request: SQOOP-737 Give user option to set number of extractors
and loaders
Posted by Bilung Lee <bl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8339/#review14027
-----------------------------------------------------------
Ship it!
Ship It!
- Bilung Lee
On Dec. 4, 2012, 7:37 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8339/
> -----------------------------------------------------------
>
> (Updated Dec. 4, 2012, 7:37 p.m.)
>
>
> Review request for Sqoop.
>
>
> Description
> -------
>
> I've implemented end-to-end behaviour to cover setting number of extractors and loaders. Current default is to have 10 extractors (=mappers) and equal number of loaders as extractors (= no reduce phase). Setting any number of loaders will add reduce phase.
>
>
> This addresses bug SQOOP-737.
> https://issues.apache.org/jira/browse/SQOOP-737
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/sqoop/common/MutableContext.java 238bbfd4d621bd0b091a5a1850d7926da6431524
> common/src/main/java/org/apache/sqoop/common/MutableMapContext.java cd9d3e3f9a6a79cb27bf21fd605331727d60b09e
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java a6d3b52fed2fd7a3f34e787603a3feeb1a0c6fd8
> connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java d5db190f831d977f4414fb0e27498885dbd03636
> core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 0cd696982772e9dd80e7584435cd5be0106bfd97
> core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java fb6b6a94722baf5c29292d72ed8908cc1e2140d7
> core/src/main/java/org/apache/sqoop/framework/configuration/ImportJobConfiguration.java 8c4dcf168a8df70b727d58e108d3df987e643bf3
> core/src/main/java/org/apache/sqoop/framework/configuration/ThrottlingForm.java PRE-CREATION
> core/src/main/resources/framework-resources.properties 019f5ca8581e466e8187549ccaa7e3f2efd74dbf
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3248e7713388f990c47261cf77d65ce46444a0b4
> execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java d899fcef662ed69b567762e55c40840c66fc2387
> execution/mapreduce/src/main/java/org/apache/sqoop/job/MapreduceExecutionError.java 30956f30cb4eb29b5c2a917c80be99a6696677e8
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 8fcdc99daa08a02775ed2ba0708eccfdfdbe078b
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java 21a2be9d7ca4fa5d3ff9dc95ad7a50c622560691
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 745a3a42bff9d74a2d62438ff0cbcc8babb00e26
> spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java 3a525c4fed5fdc8372f83b72c5a37538955618b8
> submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java a64a477577b84afb0d0848516186682dda9b6310
>
> Diff: https://reviews.apache.org/r/8339/diff/
>
>
> Testing
> -------
>
> Unit tests are passing and I've tested it on real cluster.
>
>
> Thanks,
>
> Jarek Cecho
>
>
Re: Review Request: SQOOP-737 Give user option to set number of extractors
and loaders
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8339/
-----------------------------------------------------------
(Updated Dec. 4, 2012, 7:37 p.m.)
Review request for Sqoop.
Changes
-------
I've rebased my patch on current head of Sqoop2 branch.
Description
-------
I've implemented end-to-end behaviour to cover setting number of extractors and loaders. Current default is to have 10 extractors (=mappers) and equal number of loaders as extractors (= no reduce phase). Setting any number of loaders will add reduce phase.
This addresses bug SQOOP-737.
https://issues.apache.org/jira/browse/SQOOP-737
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/common/MutableContext.java 238bbfd4d621bd0b091a5a1850d7926da6431524
common/src/main/java/org/apache/sqoop/common/MutableMapContext.java cd9d3e3f9a6a79cb27bf21fd605331727d60b09e
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java a6d3b52fed2fd7a3f34e787603a3feeb1a0c6fd8
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java d5db190f831d977f4414fb0e27498885dbd03636
core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 0cd696982772e9dd80e7584435cd5be0106bfd97
core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java fb6b6a94722baf5c29292d72ed8908cc1e2140d7
core/src/main/java/org/apache/sqoop/framework/configuration/ImportJobConfiguration.java 8c4dcf168a8df70b727d58e108d3df987e643bf3
core/src/main/java/org/apache/sqoop/framework/configuration/ThrottlingForm.java PRE-CREATION
core/src/main/resources/framework-resources.properties 019f5ca8581e466e8187549ccaa7e3f2efd74dbf
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3248e7713388f990c47261cf77d65ce46444a0b4
execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java d899fcef662ed69b567762e55c40840c66fc2387
execution/mapreduce/src/main/java/org/apache/sqoop/job/MapreduceExecutionError.java 30956f30cb4eb29b5c2a917c80be99a6696677e8
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 8fcdc99daa08a02775ed2ba0708eccfdfdbe078b
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java 21a2be9d7ca4fa5d3ff9dc95ad7a50c622560691
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 745a3a42bff9d74a2d62438ff0cbcc8babb00e26
spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java 3a525c4fed5fdc8372f83b72c5a37538955618b8
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java a64a477577b84afb0d0848516186682dda9b6310
Diff: https://reviews.apache.org/r/8339/diff/
Testing
-------
Unit tests are passing and I've tested it on real cluster.
Thanks,
Jarek Cecho