You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2008/10/07 07:42:44 UTC
[jira] Created: (HADOOP-4360) Reducers hang in SHUFFLING phase due
to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
Reducers hang in SHUFFLING phase due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
---------------------------------------------------------------------------------------------------------
Key: HADOOP-4360
URL: https://issues.apache.org/jira/browse/HADOOP-4360
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.17.2
Reporter: Zheng Shao
On our cluster we have seen JobTracker went to a weird state that a lot of TaskTrackers are getting duplicate entries in TaskTracker.FetchStatus.allMapEvents.
Since TaskTracker fetches new completed map tasks using the size of the allMapEvents as starting index, this prohibits the tasktracker from getting all completed map tasks. And as a result, reducer just hangs in the shuffling status.
The problem does not get fixed by killing and restarting TaskTracker, and when it happens a lot of TaskTrackers will show the same problem.
It seems some problems happen to the communication between JobTracker and TaskTracker.
An easy preventive fix will be to include the starting index of the list of completed map tasks from JobTracker to TaskTracker, so that TaskTracker can just throw away the data if the starting index does not match the current size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4360) Reducers hang in SHUFFLING phase
due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637597#action_12637597 ]
Arun C Murthy commented on HADOOP-4360:
---------------------------------------
Ok, thanks for the clarification - it's good to know that the problem lies at the TaskTracker end.
Any idea _why_ the TaskTracker is getting messed up?
> Reducers hang in SHUFFLING phase due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4360
> URL: https://issues.apache.org/jira/browse/HADOOP-4360
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Zheng Shao
>
> On our cluster we have seen JobTracker went to a weird state that a lot of TaskTrackers are getting duplicate entries in TaskTracker.FetchStatus.allMapEvents.
> Since TaskTracker fetches new completed map tasks using the size of the allMapEvents as starting index, this prohibits the tasktracker from getting all completed map tasks. And as a result, reducer just hangs in the shuffling status.
> The problem does not get fixed by killing and restarting TaskTracker, and when it happens a lot of TaskTrackers will show the same problem.
> It seems some problems happen to the communication between JobTracker and TaskTracker.
> An easy preventive fix will be to include the starting index of the list of completed map tasks from JobTracker to TaskTracker, so that TaskTracker can just throw away the data if the starting index does not match the current size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4360) Reducers hang in SHUFFLING phase
due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637980#action_12637980 ]
Joydeep Sen Sarma commented on HADOOP-4360:
-------------------------------------------
this looks like a dup of 3155. we haven't applied the patch yet (I think) - but i did just now upload an attachment to 3155 with the output of hadoop job -events. please let us know if we need to run any additional commands.
> Reducers hang in SHUFFLING phase due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4360
> URL: https://issues.apache.org/jira/browse/HADOOP-4360
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Zheng Shao
>
> On our cluster we have seen JobTracker went to a weird state that a lot of TaskTrackers are getting duplicate entries in TaskTracker.FetchStatus.allMapEvents.
> Since TaskTracker fetches new completed map tasks using the size of the allMapEvents as starting index, this prohibits the tasktracker from getting all completed map tasks. And as a result, reducer just hangs in the shuffling status.
> The problem does not get fixed by killing and restarting TaskTracker, and when it happens a lot of TaskTrackers will show the same problem.
> It seems some problems happen to the communication between JobTracker and TaskTracker.
> An easy preventive fix will be to include the starting index of the list of completed map tasks from JobTracker to TaskTracker, so that TaskTracker can just throw away the data if the starting index does not match the current size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4360) Reducers hang in SHUFFLING phase
due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637638#action_12637638 ]
Arun C Murthy commented on HADOOP-4360:
---------------------------------------
bq. The problem can easily happen if there are multiple requests to the job tracker at the same time: [...]
But that should not happen at all... each job has a single FetchStatus which maintains the fromEventId.
I guess we should look closer if we are mucking up there... thoughts?
> Reducers hang in SHUFFLING phase due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4360
> URL: https://issues.apache.org/jira/browse/HADOOP-4360
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Zheng Shao
>
> On our cluster we have seen JobTracker went to a weird state that a lot of TaskTrackers are getting duplicate entries in TaskTracker.FetchStatus.allMapEvents.
> Since TaskTracker fetches new completed map tasks using the size of the allMapEvents as starting index, this prohibits the tasktracker from getting all completed map tasks. And as a result, reducer just hangs in the shuffling status.
> The problem does not get fixed by killing and restarting TaskTracker, and when it happens a lot of TaskTrackers will show the same problem.
> It seems some problems happen to the communication between JobTracker and TaskTracker.
> An easy preventive fix will be to include the starting index of the list of completed map tasks from JobTracker to TaskTracker, so that TaskTracker can just throw away the data if the starting index does not match the current size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4360) Reducers hang in SHUFFLING phase
due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637692#action_12637692 ]
Arun C Murthy commented on HADOOP-4360:
---------------------------------------
Thanks Zheng, more details would really help!
> Reducers hang in SHUFFLING phase due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4360
> URL: https://issues.apache.org/jira/browse/HADOOP-4360
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Zheng Shao
>
> On our cluster we have seen JobTracker went to a weird state that a lot of TaskTrackers are getting duplicate entries in TaskTracker.FetchStatus.allMapEvents.
> Since TaskTracker fetches new completed map tasks using the size of the allMapEvents as starting index, this prohibits the tasktracker from getting all completed map tasks. And as a result, reducer just hangs in the shuffling status.
> The problem does not get fixed by killing and restarting TaskTracker, and when it happens a lot of TaskTrackers will show the same problem.
> It seems some problems happen to the communication between JobTracker and TaskTracker.
> An easy preventive fix will be to include the starting index of the list of completed map tasks from JobTracker to TaskTracker, so that TaskTracker can just throw away the data if the starting index does not match the current size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4360) Reducers hang in SHUFFLING phase
due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637587#action_12637587 ]
Arun C Murthy commented on HADOOP-4360:
---------------------------------------
It is disturbing that the JobTracker had duplicate events. I think we should focus on fixing that for a start...
Zheng what was the output of
$ bin/hadoop job -events -events <job-id> 0 10000000
> Reducers hang in SHUFFLING phase due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4360
> URL: https://issues.apache.org/jira/browse/HADOOP-4360
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Zheng Shao
>
> On our cluster we have seen JobTracker went to a weird state that a lot of TaskTrackers are getting duplicate entries in TaskTracker.FetchStatus.allMapEvents.
> Since TaskTracker fetches new completed map tasks using the size of the allMapEvents as starting index, this prohibits the tasktracker from getting all completed map tasks. And as a result, reducer just hangs in the shuffling status.
> The problem does not get fixed by killing and restarting TaskTracker, and when it happens a lot of TaskTrackers will show the same problem.
> It seems some problems happen to the communication between JobTracker and TaskTracker.
> An easy preventive fix will be to include the starting index of the list of completed map tasks from JobTracker to TaskTracker, so that TaskTracker can just throw away the data if the starting index does not match the current size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4360) Reducers hang in SHUFFLING phase
due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637598#action_12637598 ]
Zheng Shao commented on HADOOP-4360:
------------------------------------
Arun, are there any changes to getCompletionEvents interface from 17 to trunk? Or are there any changes in the plan?
If not, I will be glad to work on a diff to add the starting index to the JobTracker's reply.
> Reducers hang in SHUFFLING phase due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4360
> URL: https://issues.apache.org/jira/browse/HADOOP-4360
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Zheng Shao
>
> On our cluster we have seen JobTracker went to a weird state that a lot of TaskTrackers are getting duplicate entries in TaskTracker.FetchStatus.allMapEvents.
> Since TaskTracker fetches new completed map tasks using the size of the allMapEvents as starting index, this prohibits the tasktracker from getting all completed map tasks. And as a result, reducer just hangs in the shuffling status.
> The problem does not get fixed by killing and restarting TaskTracker, and when it happens a lot of TaskTrackers will show the same problem.
> It seems some problems happen to the communication between JobTracker and TaskTracker.
> An easy preventive fix will be to include the starting index of the list of completed map tasks from JobTracker to TaskTracker, so that TaskTracker can just throw away the data if the starting index does not match the current size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4360) Reducers hang in SHUFFLING phase
due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637857#action_12637857 ]
Devaraj Das commented on HADOOP-4360:
-------------------------------------
This is related to HADOOP-3155. In addition to what you already have for debugging this, could you please apply the patch for the 0.17 release from HADOOP-3155 and deploy.
> Reducers hang in SHUFFLING phase due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4360
> URL: https://issues.apache.org/jira/browse/HADOOP-4360
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Zheng Shao
>
> On our cluster we have seen JobTracker went to a weird state that a lot of TaskTrackers are getting duplicate entries in TaskTracker.FetchStatus.allMapEvents.
> Since TaskTracker fetches new completed map tasks using the size of the allMapEvents as starting index, this prohibits the tasktracker from getting all completed map tasks. And as a result, reducer just hangs in the shuffling status.
> The problem does not get fixed by killing and restarting TaskTracker, and when it happens a lot of TaskTrackers will show the same problem.
> It seems some problems happen to the communication between JobTracker and TaskTracker.
> An easy preventive fix will be to include the starting index of the list of completed map tasks from JobTracker to TaskTracker, so that TaskTracker can just throw away the data if the starting index does not match the current size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4360) Reducers hang in SHUFFLING phase
due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637593#action_12637593 ]
Zheng Shao commented on HADOOP-4360:
------------------------------------
The array of completion events at the JobTracker is correct - there is no duplicate. I didn't try that -events but I have a web interface to view the completed events on the job tracker.
The problem is only at the TaskTracker. However, different TaskTrackers have different patterns of duplicates.
A lot of them (but not all) are 100 duplicates at a time (1,2,3...,100, 1,2,3,...100), which also signals the problem is at the communication protocol (where we poll 100 events at a time).
> Reducers hang in SHUFFLING phase due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4360
> URL: https://issues.apache.org/jira/browse/HADOOP-4360
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Zheng Shao
>
> On our cluster we have seen JobTracker went to a weird state that a lot of TaskTrackers are getting duplicate entries in TaskTracker.FetchStatus.allMapEvents.
> Since TaskTracker fetches new completed map tasks using the size of the allMapEvents as starting index, this prohibits the tasktracker from getting all completed map tasks. And as a result, reducer just hangs in the shuffling status.
> The problem does not get fixed by killing and restarting TaskTracker, and when it happens a lot of TaskTrackers will show the same problem.
> It seems some problems happen to the communication between JobTracker and TaskTracker.
> An easy preventive fix will be to include the starting index of the list of completed map tasks from JobTracker to TaskTracker, so that TaskTracker can just throw away the data if the starting index does not match the current size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4360) Reducers hang in SHUFFLING phase
due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637606#action_12637606 ]
Zheng Shao commented on HADOOP-4360:
------------------------------------
The problem can easily happen if there are multiple requests to the job tracker at the same time:
Let's say TaskTracker's allMapEvents contains 5 elements now, and 2 requests to the JobTracker are started at the same time, so both requests bear a starting index of 5.
The 2 requests will get identical replies from JobTracker. Then they are added to the completion events separately, which caused the duplicates.
A simple "synchronized(fromEventId)" wrapper around the "jobClient.getTaskCompletionEvents" call will avoid the problem. In case we don't want to do that, we should at least check whether fromEventId has changed during the RPC call.
(let me withdraw the idea of adding starting index to the job tracker reply - that does not seem necessary)
TaskTracker.java:(from 0.17.2)
854:
private List<TaskCompletionEvent> queryJobTracker(IntWritable fromEventId,
String jobId,
InterTrackerProtocol jobClient)
throws IOException {
TaskCompletionEvent t[] = jobClient.getTaskCompletionEvents(
jobId,
fromEventId.get(),
probe_sample_size);
//we are interested in map task completion events only. So store
//only those
List <TaskCompletionEvent> recentMapEvents =
new ArrayList<TaskCompletionEvent>();
for (int i = 0; i < t.length; i++) {
if (t[i].isMap) {
recentMapEvents.add(t[i]);
}
}
fromEventId.set(fromEventId.get() + t.length);
return recentMapEvents;
613:
public void fetchMapCompletionEvents() throws IOException {
List <TaskCompletionEvent> recentMapEvents =
queryJobTracker(fromEventId, jobId, jobClient);
synchronized (allMapEvents) {
allMapEvents.addAll(recentMapEvents);
}
}
> Reducers hang in SHUFFLING phase due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4360
> URL: https://issues.apache.org/jira/browse/HADOOP-4360
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Zheng Shao
>
> On our cluster we have seen JobTracker went to a weird state that a lot of TaskTrackers are getting duplicate entries in TaskTracker.FetchStatus.allMapEvents.
> Since TaskTracker fetches new completed map tasks using the size of the allMapEvents as starting index, this prohibits the tasktracker from getting all completed map tasks. And as a result, reducer just hangs in the shuffling status.
> The problem does not get fixed by killing and restarting TaskTracker, and when it happens a lot of TaskTrackers will show the same problem.
> It seems some problems happen to the communication between JobTracker and TaskTracker.
> An easy preventive fix will be to include the starting index of the list of completed map tasks from JobTracker to TaskTracker, so that TaskTracker can just throw away the data if the starting index does not match the current size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4360) Reducers hang in SHUFFLING phase
due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637687#action_12637687 ]
Zheng Shao commented on HADOOP-4360:
------------------------------------
Yep I agree with your reasoning. I tried to find other reasons but I couldn't. Also it's not easy to reproduce the problem.
Let me put an assert in our in-house code, and try to dump some more information when duplicate events happens.
Will put up more details when I've got more information.
> Reducers hang in SHUFFLING phase due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4360
> URL: https://issues.apache.org/jira/browse/HADOOP-4360
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Zheng Shao
>
> On our cluster we have seen JobTracker went to a weird state that a lot of TaskTrackers are getting duplicate entries in TaskTracker.FetchStatus.allMapEvents.
> Since TaskTracker fetches new completed map tasks using the size of the allMapEvents as starting index, this prohibits the tasktracker from getting all completed map tasks. And as a result, reducer just hangs in the shuffling status.
> The problem does not get fixed by killing and restarting TaskTracker, and when it happens a lot of TaskTrackers will show the same problem.
> It seems some problems happen to the communication between JobTracker and TaskTracker.
> An easy preventive fix will be to include the starting index of the list of completed map tasks from JobTracker to TaskTracker, so that TaskTracker can just throw away the data if the starting index does not match the current size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-4360) Reducers hang in SHUFFLING phase due
to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao resolved HADOOP-4360.
--------------------------------
Resolution: Duplicate
Duplicate of http://issues.apache.org/jira/browse/HADOOP-3155
> Reducers hang in SHUFFLING phase due to duplicate completed tasks in TaskTracker.FetchStatus.allMapEvents
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4360
> URL: https://issues.apache.org/jira/browse/HADOOP-4360
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Zheng Shao
>
> On our cluster we have seen JobTracker went to a weird state that a lot of TaskTrackers are getting duplicate entries in TaskTracker.FetchStatus.allMapEvents.
> Since TaskTracker fetches new completed map tasks using the size of the allMapEvents as starting index, this prohibits the tasktracker from getting all completed map tasks. And as a result, reducer just hangs in the shuffling status.
> The problem does not get fixed by killing and restarting TaskTracker, and when it happens a lot of TaskTrackers will show the same problem.
> It seems some problems happen to the communication between JobTracker and TaskTracker.
> An easy preventive fix will be to include the starting index of the list of completed map tasks from JobTracker to TaskTracker, so that TaskTracker can just throw away the data if the starting index does not match the current size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.