You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2007/04/04 07:50:32 UTC
[jira] Created: (HADOOP-1201) Progress reporting can be improved
for both Map/Reduce tasks
Progress reporting can be improved for both Map/Reduce tasks
------------------------------------------------------------
Key: HADOOP-1201
URL: https://issues.apache.org/jira/browse/HADOOP-1201
Project: Hadoop
Issue Type: Improvement
Components: mapred
Reporter: Devaraj Das
Fix For: 0.13.0
Both the map and reduce tasks do progress reporting in separate threads. However, in the ReduceTask, after the sort phase, the progress reporting happens inline with the reducer invocations. This slows down the Reduce phase since RPC is involved for every progress report. The better thing to do would be to do progress reporting for all phases in separate threads and have the tasks just update the progress fields.
One proposal is to extract out the reporting stuff that is there in MapTask/ReduceTask and put it in the Task superclass as a new class, and have methods in the new class that control what/when progress is reported. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-1201) Progress reporting can be improved
for both Map/Reduce tasks
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das resolved HADOOP-1201.
---------------------------------
Resolution: Duplicate
As Vivek mentioned, this got fixed in HADOOP-1462.
> Progress reporting can be improved for both Map/Reduce tasks
> ------------------------------------------------------------
>
> Key: HADOOP-1201
> URL: https://issues.apache.org/jira/browse/HADOOP-1201
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Vivek Ratan
>
> Both the map and reduce tasks do progress reporting in separate threads. However, in the ReduceTask, after the sort phase, the progress reporting happens inline with the reducer invocations. This slows down the Reduce phase since RPC is involved for every progress report. The better thing to do would be to do progress reporting for all phases in separate threads and have the tasks just update the progress fields.
> One proposal is to extract out the reporting stuff that is there in MapTask/ReduceTask and put it in the Task superclass as a new class, and have methods in the new class that control what/when progress is reported. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1201) Progress reporting can be improved
for both Map/Reduce tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1201:
----------------------------------
Fix Version/s: (was: 0.13.0)
> Progress reporting can be improved for both Map/Reduce tasks
> ------------------------------------------------------------
>
> Key: HADOOP-1201
> URL: https://issues.apache.org/jira/browse/HADOOP-1201
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
>
> Both the map and reduce tasks do progress reporting in separate threads. However, in the ReduceTask, after the sort phase, the progress reporting happens inline with the reducer invocations. This slows down the Reduce phase since RPC is involved for every progress report. The better thing to do would be to do progress reporting for all phases in separate threads and have the tasks just update the progress fields.
> One proposal is to extract out the reporting stuff that is there in MapTask/ReduceTask and put it in the Task superclass as a new class, and have methods in the new class that control what/when progress is reported. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1201) Progress reporting can be improved
for both Map/Reduce tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500489 ]
Owen O'Malley commented on HADOOP-1201:
---------------------------------------
But once you have made the progress a separate thread, the ping provides little value. The interface will look like:
{code}
boolean updateState(String taskid, int progressCount, float progress, String state, TaskStatus.Phase phase, Counters count);
{code}
I don't see the point of having two threads that are both calling upto the task tracker every second, especially since the ping thread is so trivial.
> Progress reporting can be improved for both Map/Reduce tasks
> ------------------------------------------------------------
>
> Key: HADOOP-1201
> URL: https://issues.apache.org/jira/browse/HADOOP-1201
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
>
> Both the map and reduce tasks do progress reporting in separate threads. However, in the ReduceTask, after the sort phase, the progress reporting happens inline with the reducer invocations. This slows down the Reduce phase since RPC is involved for every progress report. The better thing to do would be to do progress reporting for all phases in separate threads and have the tasks just update the progress fields.
> One proposal is to extract out the reporting stuff that is there in MapTask/ReduceTask and put it in the Task superclass as a new class, and have methods in the new class that control what/when progress is reported. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1201) Progress reporting can be improved
for both Map/Reduce tasks
Posted by "Vivek Ratan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vivek Ratan reassigned HADOOP-1201:
-----------------------------------
Assignee: Vivek Ratan
> Progress reporting can be improved for both Map/Reduce tasks
> ------------------------------------------------------------
>
> Key: HADOOP-1201
> URL: https://issues.apache.org/jira/browse/HADOOP-1201
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Vivek Ratan
>
> Both the map and reduce tasks do progress reporting in separate threads. However, in the ReduceTask, after the sort phase, the progress reporting happens inline with the reducer invocations. This slows down the Reduce phase since RPC is involved for every progress report. The better thing to do would be to do progress reporting for all phases in separate threads and have the tasks just update the progress fields.
> One proposal is to extract out the reporting stuff that is there in MapTask/ReduceTask and put it in the Task superclass as a new class, and have methods in the new class that control what/when progress is reported. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1201) Progress reporting can be improved
for both Map/Reduce tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496070 ]
Owen O'Malley commented on HADOOP-1201:
---------------------------------------
I think an interesting way of doing it would be to have the "ping" thread sending status instead. So the Reporter would update the values in memory and the ping thread would send them up once a second. The status would include a counter that would get incremented when the application called progress. The task tracker would then detect differences to determine if the application is making progress.
> Progress reporting can be improved for both Map/Reduce tasks
> ------------------------------------------------------------
>
> Key: HADOOP-1201
> URL: https://issues.apache.org/jira/browse/HADOOP-1201
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
>
> Both the map and reduce tasks do progress reporting in separate threads. However, in the ReduceTask, after the sort phase, the progress reporting happens inline with the reducer invocations. This slows down the Reduce phase since RPC is involved for every progress report. The better thing to do would be to do progress reporting for all phases in separate threads and have the tasks just update the progress fields.
> One proposal is to extract out the reporting stuff that is there in MapTask/ReduceTask and put it in the Task superclass as a new class, and have methods in the new class that control what/when progress is reported. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1201) Progress reporting can be improved
for both Map/Reduce tasks
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496097 ]
Devaraj Das commented on HADOOP-1201:
-------------------------------------
Makes sense. Maybe HADOOP-1235 should be merged with this issue.
> Progress reporting can be improved for both Map/Reduce tasks
> ------------------------------------------------------------
>
> Key: HADOOP-1201
> URL: https://issues.apache.org/jira/browse/HADOOP-1201
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
>
> Both the map and reduce tasks do progress reporting in separate threads. However, in the ReduceTask, after the sort phase, the progress reporting happens inline with the reducer invocations. This slows down the Reduce phase since RPC is involved for every progress report. The better thing to do would be to do progress reporting for all phases in separate threads and have the tasks just update the progress fields.
> One proposal is to extract out the reporting stuff that is there in MapTask/ReduceTask and put it in the Task superclass as a new class, and have methods in the new class that control what/when progress is reported. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1201) Progress reporting can be improved
for both Map/Reduce tasks
Posted by "Vivek Ratan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499781 ]
Vivek Ratan commented on HADOOP-1201:
-------------------------------------
I'm assuming that what's suggested here is that the Ping thread just make one RPC call to the TaskTracker, which would support both the ping and progress functionalities.
The ping call and the progress call have different signatures, return values, and meaning. They may also be called at different frequencies (the task pings every second as long as it is alive, whereas it may not always send a progress report at the same frequency, especially when it's not made any progress since the last call, but it's still chugging along).
So, IMO, unless there's a signifcant performance penalty in having two RPC calls from the task to the TaskTracker, combining the two calls will make the code difficult to understand. The return value and the parameters for the combined call will really apply to two separate functions (ping and progress) and the code will be messier. Do we see a performance hit? If not, maybe we shouldn't use the ping thread to send the progress.
If we do decide to combine the calls, we should eliminate TaskUmbilicalProtocol::Ping(). TaskUmbilicalProtocol::progress() should return a boolean value (false if the task was not foudn in the TaskTracker, true otherwise). If the RPC fails (the TT is dead) or returns false, the Task kills itself. This is also what Devaraj suggested in HADOOP-1235.
My recommendation:
1. We leave the ping and progress threads as they are. If we see performance issues with two RPC calls, we eliminate Ping() and modify progress() to return a boolean.
2. Back to the original bug: there is inline code in ReduceTask to report progress. Moving this code to a separate thread, as we do for MapTask, is probably the right thing to do, but it ties in with whatever solution we adopt for HADOOP-1431. Please see my comments there.
> Progress reporting can be improved for both Map/Reduce tasks
> ------------------------------------------------------------
>
> Key: HADOOP-1201
> URL: https://issues.apache.org/jira/browse/HADOOP-1201
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
>
> Both the map and reduce tasks do progress reporting in separate threads. However, in the ReduceTask, after the sort phase, the progress reporting happens inline with the reducer invocations. This slows down the Reduce phase since RPC is involved for every progress report. The better thing to do would be to do progress reporting for all phases in separate threads and have the tasks just update the progress fields.
> One proposal is to extract out the reporting stuff that is there in MapTask/ReduceTask and put it in the Task superclass as a new class, and have methods in the new class that control what/when progress is reported. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.