[jira] [Commented] (MAPREDUCE-5932) Provide an option to use a dedicated reduce-side shuffle log

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232221#comment-14232221 ] 

Jason Lowe commented on MAPREDUCE-5932:
---------------------------------------

Thanks for updating the patch, Gera.  Looks good overall.  Gave the patch a test drive, and I noticed that some of the logs that occur during shuffle are not being redirected to the shuffle log, specifically org.apache.hadoop.mapred.Merger.  MergeManagerImpl is in the shuffle log but the Merger logs are not.  IMHO it'd be nice if the Merger logs also appeared in the separate shuffle log since those logs occur during the shuffle phase since we merge while shuffling.

> Provide an option to use a dedicated reduce-side shuffle log
> ------------------------------------------------------------
>
>                 Key: MAPREDUCE-5932
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5932
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 2.4.0
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5932.v01.patch, MAPREDUCE-5932.v02.patch, MAPREDUCE-5932.v03.patch, MAPREDUCE-5932.v04.patch, MAPREDUCE-5932.v05.patch
>
>
> For reducers in large jobs our users cannot easily spot portions of the log associated with problems with their code. An example reducer with INFO-level logging generates ~3500 lines / ~700KiB  lines per second. 95% of the log is the client-side of the shuffle {{org.apache.hadoop.mapreduce.task.reduce.*}}
> {code}
> $ wc syslog 
>     3642   48192  691013 syslog
> $ grep task.reduce syslog | wc 
>     3424   46534  659038
> $ grep task.reduce.ShuffleScheduler syslog | wc 
>     1521   17745  251458
> $ grep task.reduce.Fetcher syslog | wc 
>     1045   15340  223683
> $ grep task.reduce.InMemoryMapOutput syslog | wc 
>      400    4800   72060
> $ grep task.reduce.MergeManagerImpl syslog | wc 
>      432    8200  106555
> {code}
> Byte percentage breakdown:
> {code}
> Shuffle total:           95%
> ShuffleScheduler:        36%
> Fetcher:                 32%
> InMemoryMapOutput:       10%
> MergeManagerImpl:        15%
> {code}
> While this is information is actually often useful for devops debugging shuffle performance issues, the job users are often lost. 
> We propose to have a dedicated syslog.shuffle file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)