[jira] Commented: (HADOOP-1485) Metrics should be there for reporting shuffle failures/successes

["David Bowen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12507489 ] 

David Bowen commented on HADOOP-1485:
-------------------------------------


Reviewing 1485.1.patch.

Two classes with the name ShuffleMetrics is confusing.  Please rename at least one of them, and add some per-class comments explaining their purposes.

TaskTracker.ShuffleMetrics:

   * shuffle_handler_busy_percent seems to be an absolute value, i.e. it should be using setMetric rather than incrMetric.  Also, shuffle_failed_outputs and shuffle_success_outputs seem to be relative values, and so should be using incrMetric rather than setMetric.
   * It may be an unnecessary optimization, but it couldn't hurt to move the shuffleMetricsRecord.update call out of the synchronized block.  update() has to do a little bit of work, and there's no need to be holding the lock.
   * MapOutputServlet is missing indentation under the first "try {".
   * The final finally may need to call shuffleMetrics.update.

ReduceTask.ReduceCopier.ShuffleMetrics:

   * I think incrMetric should be being used for shuffle_failed_fetches and shuffle_success_fetches.
   * Same comment about moving the shuffleMetrics.update() call out of the synchronized block.




> Metrics should be there for reporting shuffle failures/successes
> ----------------------------------------------------------------
>
>                 Key: HADOOP-1485
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1485
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.14.0
>
>         Attachments: 1485.1.patch, shuffle-metrics.patch
>
>
> It would be nice to have metrics for the shuffle phase which reports the failures/successes for the fetches. This would aid in performance tests and in debugging (shuffle).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.