You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Aaron Kimball (JIRA)" <ji...@apache.org> on 2010/01/09 00:34:54 UTC
[jira] Created: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
LocalJobRunner should support parallel mapper execution
-------------------------------------------------------
Key: MAPREDUCE-1367
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Aaron Kimball
Assignee: Aaron Kimball
The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802036#action_12802036 ]
Jeff Hammerbacher commented on MAPREDUCE-1367:
----------------------------------------------
What is the delay in committing this patch?
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Status: Patch Available (was: Open)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806585#action_12806585 ]
Hadoop QA commented on MAPREDUCE-1367:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12431823/MAPREDUCE-1367.7.patch
against trunk revision 904609.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/console
This message is automatically generated.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Status: Open (was: Patch Available)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Attachment: MAPREDUCE-1367.5.patch
Todd, good catch: it didn't handle the 0-mapper case. It does now. Testcase included.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Status: Patch Available (was: Open)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball reassigned MAPREDUCE-1367:
----------------------------------------
Assignee: Aaron Kimball (was: ZhuGuanyin)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805390#action_12805390 ]
Aaron Kimball commented on MAPREDUCE-1367:
------------------------------------------
Test failures are all unrelated (TT blacklisting has now falied two tests in a row?)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805389#action_12805389 ]
Hadoop QA commented on MAPREDUCE-1367:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12431509/MAPREDUCE-1367.6.patch
against trunk revision 903508.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/console
This message is automatically generated.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Attachment: MAPREDUCE-1367.7.patch
new patch, sync'd with trunk.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Status: Patch Available (was: Open)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799570#action_12799570 ]
Hadoop QA commented on MAPREDUCE-1367:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12430081/MAPREDUCE-1367.5.patch
against trunk revision 898486.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/console
This message is automatically generated.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799069#action_12799069 ]
Aaron Kimball commented on MAPREDUCE-1367:
------------------------------------------
Tests that failed in Hudson all pass locally. These tests don't appear to use the LocalJobRunner.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806590#action_12806590 ]
Aaron Kimball commented on MAPREDUCE-1367:
------------------------------------------
Test failures seem unrelated - classpath issues on hudson again?
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798299#action_12798299 ]
Hadoop QA commented on MAPREDUCE-1367:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12429800/MAPREDUCE-1367.patch
against trunk revision 897118.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/console
This message is automatically generated.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Status: Open (was: Patch Available)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Status: Open (was: Patch Available)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799484#action_12799484 ]
Hadoop QA commented on MAPREDUCE-1367:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12430043/MAPREDUCE-1367.4.patch
against trunk revision 898486.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/console
This message is automatically generated.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Assignee: ZhuGuanyin (was: Aaron Kimball)
Status: Patch Available (was: Open)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: ZhuGuanyin
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799448#action_12799448 ]
Todd Lipcon commented on MAPREDUCE-1367:
----------------------------------------
- MapTaskRunnable: make most of the instance variables final?
- it should be clearer that mapOutputFiles is an "out param" for getMapTaskRunnables - ie that it's expected to be intitially empty. Alternatively, it looks like you may be able to remove this variable entirely by instead storing the MapOutputFile in the MapRunnable instance, and then iterating directly over the MapRunnables in the reducer. Does that make sense? However, the way you've done it is less invasive to the reduce side, so if you don't see the benefit, feel free to ignore this suggestion.
- Does this handle the degenerate case of 0-map jobs? It sounds ridiculous, but I recall previous JIRAs for this situation, since occasionally people have a cron job that periodically processes a given directory. If the directory is empty, it may generate a job with no input splits and thus no tasks.
Aside from that, looks good to me.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828467#action_12828467 ]
Hudson commented on MAPREDUCE-1367:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/])
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Attachment: MAPREDUCE-1367.3.patch
new patch addresses findbugs warnings.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Attachment: MAPREDUCE-1367.patch
Attaching a patch that implements this improvement. This patch includes a test case which launches 6 mappers concurrently; these mappers run on a variety of schedules (some are faster, some are slower) in an attempt to suss out any race conditions that might develop.
The level of parallelism is controlled by a new parameter: {{mapred.local.map.tasks.maximum}}. This defaults to 1, so that unspecified behavior is as before.
I also tested this by running the 'pi' example from the command line:
{code}bin/hadoop jar hadoop-mapred-examples-0.22.0-SNAPSHOT.jar pi -D mapreduce.jobtracker.address=local -D mapreduce.local.map.tasks.maximum=2 20 5000000
{code}
With {{mapreduce.local.map.tasks.maximum}} set to 1, this takes 13.5 seconds on my machine. With it set to 2 or above (I have two cores), the runtime drops to 8.5 seconds.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Status: Patch Available (was: Open)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Status: Patch Available (was: Open)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Status: Patch Available (was: Open)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Attachment: MAPREDUCE-1367.4.patch
Chris, all those suggestions make sense. Here's a new patch that implements them.
I also added a static getter/setter API for the config param
{code}
public static void LocalJobRunner.setLocalMaxRunningMaps(JobContext, int)
public static int LocalJobRunner.getLocalMaxRunningMaps(JobContext)
{code}
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Status: Open (was: Patch Available)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Attachment: MAPREDUCE-1367.2.patch
New patch; rewrites counter handling to make TestJobCounters pass.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1367:
-------------------------------------
Status: Open (was: Patch Available)
As a first pass:
* {{map_tasks}} is volatile, but increment/decrement is not atomic. Consider {{AtomicInteger}} instead
* {{getMapExecutor}} should be named {{createMapExecutor}} ; adding tasks to the ExecutorService belongs at the caller. The logging should probably be at DEBUG instead of INFO
* {{storedException}} should probably be Throwable instead of Exception
* if {{LOCAL_MAX_MAPS}} is < 1, that should be an error
Otherwise, this looks pretty good.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Status: Open (was: Patch Available)
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1367:
-------------------------------------
Attachment: MAPREDUCE-1367.6.patch
new patch, sync'd with trunk.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799042#action_12799042 ]
Hadoop QA commented on MAPREDUCE-1367:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12429958/MAPREDUCE-1367.2.patch
against trunk revision 898019.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/console
This message is automatically generated.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805302#action_12805302 ]
Tom White commented on MAPREDUCE-1367:
--------------------------------------
Aaron, this patch has gone stale. Could you regenerate, then I think it's ready to be committed.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1367:
---------------------------------
Resolution: Fixed
Fix Version/s: 0.22.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Aaron!
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support
parallel mapper execution
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806693#action_12806693 ]
Hudson commented on MAPREDUCE-1367:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #221 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/])
. LocalJobRunner should support parallel mapper execution. Contributed by Aaron Kimball.
> LocalJobRunner should support parallel mapper execution
> -------------------------------------------------------
>
> Key: MAPREDUCE-1367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch
>
>
> The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.