You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Runping Qi (JIRA)" <ji...@apache.org> on 2008/08/25 19:53:44 UTC
[jira] Created: (HADOOP-4015) Need counters for spils and merges in
mappers and reducers.
Need counters for spils and merges in mappers and reducers.
-----------------------------------------------------------
Key: HADOOP-4015
URL: https://issues.apache.org/jira/browse/HADOOP-4015
Project: Hadoop Core
Issue Type: Bug
Reporter: Runping Qi
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4015) Need counters for spils and merges
in mappers and reducers.
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625474#action_12625474 ]
Viraj Bhat commented on HADOOP-4015:
------------------------------------
+1 presently I have to analyze the job tracker logs and indirectly calculate these parameters through Local Bytes Read, Local Bytes Written, Map and Reduce Input Output bytes.
> Need counters for spils and merges in mappers and reducers.
> -----------------------------------------------------------
>
> Key: HADOOP-4015
> URL: https://issues.apache.org/jira/browse/HADOOP-4015
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
>
> Currently, it is hard to know whether a mapper did spills and how many if it did.
> It is even harder to figure out how many merges happened in a mapper or a reducer.
> Those data should be made available through counters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4015) Need counters for spils and merges
in mappers and reducers.
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625937#action_12625937 ]
Runping Qi commented on HADOOP-4015:
------------------------------------
Also, it is desirable to log these counters periodically, because, the counters per task are not available until the task complete.
However, it is helpful to know the progress of the spills/merges if they are huge.
> Need counters for spils and merges in mappers and reducers.
> -----------------------------------------------------------
>
> Key: HADOOP-4015
> URL: https://issues.apache.org/jira/browse/HADOOP-4015
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
>
> Currently, it is hard to know whether a mapper did spills and how many if it did.
> It is even harder to figure out how many merges happened in a mapper or a reducer.
> Those data should be made available through counters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-4015) Need counters for spils and merges
in mappers and reducers.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas resolved HADOOP-4015.
-----------------------------------
Resolution: Duplicate
This looks like a duplicate of HADOOP-2774
> Need counters for spils and merges in mappers and reducers.
> -----------------------------------------------------------
>
> Key: HADOOP-4015
> URL: https://issues.apache.org/jira/browse/HADOOP-4015
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
>
> Currently, it is hard to know whether a mapper did spills and how many if it did.
> It is even harder to figure out how many merges happened in a mapper or a reducer.
> Those data should be made available through counters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4015) Need counters for spils and merges in
mappers and reducers.
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Runping Qi updated HADOOP-4015:
-------------------------------
Component/s: mapred
Description:
Currently, it is hard to know whether a mapper did spills and how many if it did.
It is even harder to figure out how many merges happened in a mapper or a reducer.
Those data should be made available through counters.
> Need counters for spils and merges in mappers and reducers.
> -----------------------------------------------------------
>
> Key: HADOOP-4015
> URL: https://issues.apache.org/jira/browse/HADOOP-4015
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
>
> Currently, it is hard to know whether a mapper did spills and how many if it did.
> It is even harder to figure out how many merges happened in a mapper or a reducer.
> Those data should be made available through counters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.