You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Devaraj K (JIRA)" <ji...@apache.org> on 2011/05/18 17:49:48 UTC
[jira] [Created] (MAPREDUCE-2512) wait(5000) and notify() mechanism
can be implemented instead of sleep(5000) in reduce task when there are no
copies in progress and no new copies to schedule
wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-2512
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task
Affects Versions: 0.20.2
Reporter: Devaraj K
Assignee: Devaraj K
{code:title=ReduceTask.java|borderStyle=solid}
try {
if (numInFlight == 0 && numScheduled == 0) {
// we should indicate progress as we don't want TT to think
// we're stuck and kill us
reporter.progress();
Thread.sleep(5000);
}
} catch (InterruptedException e) { } // IGNORE
{code}
Here if we have no copies in flight and we can't schedule anything new, it is going to wait for 5000 millis. Instead of waiting for 5000 millis, this thread can wait with timeout and GetMapEventsThread can notify it if gets new map completion events earlier than 5000 millis time.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2512) wait(5000) and notify()
mechanism can be implemented instead of sleep(5000) in reduce task when
there are no copies in progress and no new copies to schedule
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043375#comment-13043375 ]
Hadoop QA commented on MAPREDUCE-2512:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12481348/MAPREDUCE-2512.patch
against trunk revision 1130994.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/343//console
This message is automatically generated.
> wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Devaraj K
> Assignee: Devaraj K
> Fix For: 0.20.4
>
> Attachments: MAPREDUCE-2512.patch
>
>
> {code:title=ReduceTask.java|borderStyle=solid}
> try {
> if (numInFlight == 0 && numScheduled == 0) {
> // we should indicate progress as we don't want TT to think
> // we're stuck and kill us
> reporter.progress();
> Thread.sleep(5000);
> }
> } catch (InterruptedException e) { } // IGNORE
> {code}
> Here if we have no copies in flight and we can't schedule anything new, it is going to wait for 5000 millis. Instead of waiting for 5000 millis, this thread can wait with timeout and GetMapEventsThread can notify it if gets new map completion events earlier than 5000 millis time.
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2512) wait(5000) and notify() mechanism
can be implemented instead of sleep(5000) in reduce task when there are no
copies in progress and no new copies to schedule
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2512:
-----------------------------------
Status: Open (was: Patch Available)
> wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Devaraj K
> Assignee: Devaraj K
> Fix For: 0.20.4
>
> Attachments: MAPREDUCE-2512.patch
>
>
> {code:title=ReduceTask.java|borderStyle=solid}
> try {
> if (numInFlight == 0 && numScheduled == 0) {
> // we should indicate progress as we don't want TT to think
> // we're stuck and kill us
> reporter.progress();
> Thread.sleep(5000);
> }
> } catch (InterruptedException e) { } // IGNORE
> {code}
> Here if we have no copies in flight and we can't schedule anything new, it is going to wait for 5000 millis. Instead of waiting for 5000 millis, this thread can wait with timeout and GetMapEventsThread can notify it if gets new map completion events earlier than 5000 millis time.
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2512) wait(5000) and notify()
mechanism can be implemented instead of sleep(5000) in reduce task when
there are no copies in progress and no new copies to schedule
Posted by "Devaraj K (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K resolved MAPREDUCE-2512.
----------------------------------
Resolution: Won't Fix
Assignee: (was: Devaraj K)
It doesn't exist in the trunk hence closing it.
> wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Devaraj K
> Attachments: MAPREDUCE-2512.patch
>
>
> {code:title=ReduceTask.java|borderStyle=solid}
> try {
> if (numInFlight == 0 && numScheduled == 0) {
> // we should indicate progress as we don't want TT to think
> // we're stuck and kill us
> reporter.progress();
> Thread.sleep(5000);
> }
> } catch (InterruptedException e) { } // IGNORE
> {code}
> Here if we have no copies in flight and we can't schedule anything new, it is going to wait for 5000 millis. Instead of waiting for 5000 millis, this thread can wait with timeout and GetMapEventsThread can notify it if gets new map completion events earlier than 5000 millis time.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2512) wait(5000) and notify()
mechanism can be implemented instead of sleep(5000) in reduce task when
there are no copies in progress and no new copies to schedule
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035559#comment-13035559 ]
Allen Wittenauer commented on MAPREDUCE-2512:
---------------------------------------------
Could this try/catch block be related to the stuck tasks we sometimes see? What sort of exceptions are InterruptedException?
> wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Devaraj K
> Assignee: Devaraj K
>
> {code:title=ReduceTask.java|borderStyle=solid}
> try {
> if (numInFlight == 0 && numScheduled == 0) {
> // we should indicate progress as we don't want TT to think
> // we're stuck and kill us
> reporter.progress();
> Thread.sleep(5000);
> }
> } catch (InterruptedException e) { } // IGNORE
> {code}
> Here if we have no copies in flight and we can't schedule anything new, it is going to wait for 5000 millis. Instead of waiting for 5000 millis, this thread can wait with timeout and GetMapEventsThread can notify it if gets new map completion events earlier than 5000 millis time.
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2512) wait(5000) and notify()
mechanism can be implemented instead of sleep(5000) in reduce task when
there are no copies in progress and no new copies to schedule
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036751#comment-13036751 ]
Devaraj K commented on MAPREDUCE-2512:
--------------------------------------
This try/catch is not related to the any stuck tasks. If there are no map outputs ready to copy, this thread goes to sleep state for 5000 millis and then continue execution after timeout. When this thread is in sleep state, if any thread performs some operation (like interrupt (), stop ()) on this thread, it will throw InterruptedException and comes out from the sleep state. After getting InterruptedException, it will ignore and continue execution normally.
This thread waits for 5000 millis even If it gets new map completion events before 5000 millis. This can be optimized such that,
{code:title=ReduceTask.java|borderStyle=solid}
reporter.progress();
- Thread.sleep(5000);
+ synchronized (lockCopy) {
+ lockCopy.wait(5000);
+ }
}
} catch (InterruptedException e) { } // IGNORE
int numNewMaps = getMapCompletionEvents();
if (numNewMaps > 0) {
+ synchronized (lockCopy) {
+ lockCopy.notify();
+ }
LOG.info(reduceTask.getTaskID() + ": " +
"Got " + numNewMaps + " new map-outputs");
{code}
> wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Devaraj K
> Assignee: Devaraj K
>
> {code:title=ReduceTask.java|borderStyle=solid}
> try {
> if (numInFlight == 0 && numScheduled == 0) {
> // we should indicate progress as we don't want TT to think
> // we're stuck and kill us
> reporter.progress();
> Thread.sleep(5000);
> }
> } catch (InterruptedException e) { } // IGNORE
> {code}
> Here if we have no copies in flight and we can't schedule anything new, it is going to wait for 5000 millis. Instead of waiting for 5000 millis, this thread can wait with timeout and GetMapEventsThread can notify it if gets new map completion events earlier than 5000 millis time.
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2512) wait(5000) and notify() mechanism
can be implemented instead of sleep(5000) in reduce task when there are no
copies in progress and no new copies to schedule
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-2512:
---------------------------------
Fix Version/s: 0.20.4
Status: Patch Available (was: Open)
Patch is provided for 0.20.4 version.
> wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Devaraj K
> Assignee: Devaraj K
> Fix For: 0.20.4
>
> Attachments: MAPREDUCE-2512.patch
>
>
> {code:title=ReduceTask.java|borderStyle=solid}
> try {
> if (numInFlight == 0 && numScheduled == 0) {
> // we should indicate progress as we don't want TT to think
> // we're stuck and kill us
> reporter.progress();
> Thread.sleep(5000);
> }
> } catch (InterruptedException e) { } // IGNORE
> {code}
> Here if we have no copies in flight and we can't schedule anything new, it is going to wait for 5000 millis. Instead of waiting for 5000 millis, this thread can wait with timeout and GetMapEventsThread can notify it if gets new map completion events earlier than 5000 millis time.
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2512) wait(5000) and notify()
mechanism can be implemented instead of sleep(5000) in reduce task when
there are no copies in progress and no new copies to schedule
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045101#comment-13045101 ]
Todd Lipcon commented on MAPREDUCE-2512:
----------------------------------------
As this is an optimization, you should prepare a patch against trunk if the issue exists there. We don't generally put optimizations backward into prior releases.
> wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Devaraj K
> Assignee: Devaraj K
> Fix For: 0.20.4
>
> Attachments: MAPREDUCE-2512.patch
>
>
> {code:title=ReduceTask.java|borderStyle=solid}
> try {
> if (numInFlight == 0 && numScheduled == 0) {
> // we should indicate progress as we don't want TT to think
> // we're stuck and kill us
> reporter.progress();
> Thread.sleep(5000);
> }
> } catch (InterruptedException e) { } // IGNORE
> {code}
> Here if we have no copies in flight and we can't schedule anything new, it is going to wait for 5000 millis. Instead of waiting for 5000 millis, this thread can wait with timeout and GetMapEventsThread can notify it if gets new map completion events earlier than 5000 millis time.
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2512) wait(5000) and notify() mechanism
can be implemented instead of sleep(5000) in reduce task when there are no
copies in progress and no new copies to schedule
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-2512:
---------------------------------
Attachment: MAPREDUCE-2512.patch
> wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-2512.patch
>
>
> {code:title=ReduceTask.java|borderStyle=solid}
> try {
> if (numInFlight == 0 && numScheduled == 0) {
> // we should indicate progress as we don't want TT to think
> // we're stuck and kill us
> reporter.progress();
> Thread.sleep(5000);
> }
> } catch (InterruptedException e) { } // IGNORE
> {code}
> Here if we have no copies in flight and we can't schedule anything new, it is going to wait for 5000 millis. Instead of waiting for 5000 millis, this thread can wait with timeout and GetMapEventsThread can notify it if gets new map completion events earlier than 5000 millis time.
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira