You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2009/01/27 09:35:02 UTC
[jira] Created: (HADOOP-5131) ReduceCopier sleeps for a hardcoded
interval of 5secs
ReduceCopier sleeps for a hardcoded interval of 5secs
-----------------------------------------------------
Key: HADOOP-5131
URL: https://issues.apache.org/jira/browse/HADOOP-5131
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.20.0
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Currently the ReduceCopier.run has a hard-coded 5s sleep which hurts jobs where latency is important, we really should have a mechanism where the thread fetching task-completion events should notify the ReduceCopier.run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5131) ReduceCopier sleeps for a hardcoded
interval of 5secs
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668073#action_12668073 ]
Devaraj Das commented on HADOOP-5131:
-------------------------------------
Looks good to me.
> ReduceCopier sleeps for a hardcoded interval of 5secs
> -----------------------------------------------------
>
> Key: HADOOP-5131
> URL: https://issues.apache.org/jira/browse/HADOOP-5131
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Attachments: HADOOP-5131_0_20090127.patch
>
>
> Currently the ReduceCopier.run has a hard-coded 5s sleep which hurts jobs where latency is important, we really should have a mechanism where the thread fetching task-completion events should notify the ReduceCopier.run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5131) ReduceCopier sleeps for a hardcoded
interval of 5secs
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668331#action_12668331 ]
Devaraj Das commented on HADOOP-5131:
-------------------------------------
Actually, I think there is a problem with the patch in MapOutputCopier.run(). The affected code is -
{code}
synchronized (scheduledCopies) {
while (scheduledCopies.isEmpty()) {
scheduledCopies.wait();
}
}
loc = scheduledCopies.remove(0);
{code}
The line {noformat}scheduledCopies.remove(0){noformat} should be inside the {noformat}synchronized(scheduledCopies){noformat} block. Otherwise there is a race condition (assume an object got inserted in the scheduledCopies list and two copier threads got notified) - thread T1 gets a notification, wakes up, but just before it does scheduledCopies.remove(0), there is a context switch to thread T2. Thread T2 also finds the scheduledCopies to be non-empty and goes ahead and makes a successful call to scheduledCopies.remove(0). Now when T1 gets to run again, it would get a null for the remove(0) call. I think we should prevent this from happening..
> ReduceCopier sleeps for a hardcoded interval of 5secs
> -----------------------------------------------------
>
> Key: HADOOP-5131
> URL: https://issues.apache.org/jira/browse/HADOOP-5131
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Attachments: HADOOP-5131_0_20090127.patch
>
>
> Currently the ReduceCopier.run has a hard-coded 5s sleep which hurts jobs where latency is important, we really should have a mechanism where the thread fetching task-completion events should notify the ReduceCopier.run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5131) ReduceCopier sleeps for a hardcoded
interval of 5secs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-5131:
----------------------------------
Attachment: HADOOP-5131_0_20090127.patch
Preliminary patch while I continue testing...
> ReduceCopier sleeps for a hardcoded interval of 5secs
> -----------------------------------------------------
>
> Key: HADOOP-5131
> URL: https://issues.apache.org/jira/browse/HADOOP-5131
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Attachments: HADOOP-5131_0_20090127.patch
>
>
> Currently the ReduceCopier.run has a hard-coded 5s sleep which hurts jobs where latency is important, we really should have a mechanism where the thread fetching task-completion events should notify the ReduceCopier.run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5131) ReduceCopier sleeps for a hardcoded
interval of 5secs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668387#action_12668387 ]
Arun C Murthy commented on HADOOP-5131:
---------------------------------------
Good catch Devaraj! I actually saw the NPE... *smile*
> ReduceCopier sleeps for a hardcoded interval of 5secs
> -----------------------------------------------------
>
> Key: HADOOP-5131
> URL: https://issues.apache.org/jira/browse/HADOOP-5131
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Attachments: HADOOP-5131_0_20090127.patch
>
>
> Currently the ReduceCopier.run has a hard-coded 5s sleep which hurts jobs where latency is important, we really should have a mechanism where the thread fetching task-completion events should notify the ReduceCopier.run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.