You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Aaron Kimball (JIRA)" <ji...@apache.org> on 2010/02/10 03:22:27 UTC
[jira] Created: (MAPREDUCE-1473) Sqoop should allow users to
control export parallelism
Sqoop should allow users to control export parallelism
------------------------------------------------------
Key: MAPREDUCE-1473
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1473
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/sqoop
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Attachments: MAPREDUCE-1473.patch
Sqoop uses MapReduce jobs to export files back to a table in the database. The degree of parallelism is controlled by the number of splits; i.e., the number of input files used. The bottleneck in the system, though, is likely to be the database itself.
Users should have the ability to tune the number of parallel exporters being used to a degree appropriate to their database deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1473) Sqoop should allow users to
control export parallelism
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832215#action_12832215 ]
Aaron Kimball commented on MAPREDUCE-1473:
------------------------------------------
Failure is in pipes; unrelated
> Sqoop should allow users to control export parallelism
> ------------------------------------------------------
>
> Key: MAPREDUCE-1473
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1473
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1473.patch
>
>
> Sqoop uses MapReduce jobs to export files back to a table in the database. The degree of parallelism is controlled by the number of splits; i.e., the number of input files used. The bottleneck in the system, though, is likely to be the database itself.
> Users should have the ability to tune the number of parallel exporters being used to a degree appropriate to their database deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1473) Sqoop should allow users to
control export parallelism
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1473:
-------------------------------------
Status: Open (was: Patch Available)
> Sqoop should allow users to control export parallelism
> ------------------------------------------------------
>
> Key: MAPREDUCE-1473
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1473
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1473.2.patch, MAPREDUCE-1473.patch
>
>
> Sqoop uses MapReduce jobs to export files back to a table in the database. The degree of parallelism is controlled by the number of splits; i.e., the number of input files used. The bottleneck in the system, though, is likely to be the database itself.
> Users should have the ability to tune the number of parallel exporters being used to a degree appropriate to their database deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1473) Sqoop should allow users to
control export parallelism
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832165#action_12832165 ]
Hadoop QA commented on MAPREDUCE-1473:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12435396/MAPREDUCE-1473.patch
against trunk revision 908321.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/312/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/312/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/312/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/312/console
This message is automatically generated.
> Sqoop should allow users to control export parallelism
> ------------------------------------------------------
>
> Key: MAPREDUCE-1473
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1473
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1473.patch
>
>
> Sqoop uses MapReduce jobs to export files back to a table in the database. The degree of parallelism is controlled by the number of splits; i.e., the number of input files used. The bottleneck in the system, though, is likely to be the database itself.
> Users should have the ability to tune the number of parallel exporters being used to a degree appropriate to their database deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1473) Sqoop should allow users to
control export parallelism
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1473:
-------------------------------------
Status: Patch Available (was: Open)
> Sqoop should allow users to control export parallelism
> ------------------------------------------------------
>
> Key: MAPREDUCE-1473
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1473
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1473.patch
>
>
> Sqoop uses MapReduce jobs to export files back to a table in the database. The degree of parallelism is controlled by the number of splits; i.e., the number of input files used. The bottleneck in the system, though, is likely to be the database itself.
> Users should have the ability to tune the number of parallel exporters being used to a degree appropriate to their database deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1473) Sqoop should allow users to
control export parallelism
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1473:
-------------------------------------
Attachment: MAPREDUCE-1473.patch
Attaching a patch which provides this functionality. This uses CombineFileInputFormat to batch up Sqoop's input files into a user-defined number of splits.
As in importing, the degree of parallelism is controlled with the {{\-m}} / {{--num-mappers}} parameters.
> Sqoop should allow users to control export parallelism
> ------------------------------------------------------
>
> Key: MAPREDUCE-1473
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1473
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1473.patch
>
>
> Sqoop uses MapReduce jobs to export files back to a table in the database. The degree of parallelism is controlled by the number of splits; i.e., the number of input files used. The bottleneck in the system, though, is likely to be the database itself.
> Users should have the ability to tune the number of parallel exporters being used to a degree appropriate to their database deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1473) Sqoop should allow users to
control export parallelism
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball resolved MAPREDUCE-1473.
--------------------------------------
Resolution: Won't Fix
This is part of Sqoop which has been removed from the MapReduce project.
> Sqoop should allow users to control export parallelism
> ------------------------------------------------------
>
> Key: MAPREDUCE-1473
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1473
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1473.2.patch, MAPREDUCE-1473.patch
>
>
> Sqoop uses MapReduce jobs to export files back to a table in the database. The degree of parallelism is controlled by the number of splits; i.e., the number of input files used. The bottleneck in the system, though, is likely to be the database itself.
> Users should have the ability to tune the number of parallel exporters being used to a degree appropriate to their database deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1473) Sqoop should allow users to
control export parallelism
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1473:
-------------------------------------
Attachment: MAPREDUCE-1473.2.patch
New patch; the bugfix in MAPREDUCE-1480 for CombineFileRecordReader eliminates some hackery in the CombineShimRecordReader / LineRecordReader interaction (e.g., the dummy MapContext).
> Sqoop should allow users to control export parallelism
> ------------------------------------------------------
>
> Key: MAPREDUCE-1473
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1473
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1473.2.patch, MAPREDUCE-1473.patch
>
>
> Sqoop uses MapReduce jobs to export files back to a table in the database. The degree of parallelism is controlled by the number of splits; i.e., the number of input files used. The bottleneck in the system, though, is likely to be the database itself.
> Users should have the ability to tune the number of parallel exporters being used to a degree appropriate to their database deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.