You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2012/10/22 17:52:13 UTC

[jira] [Updated] (MAPREDUCE-4517) Too many INFO messages written out during AM to RM heartbeat

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Lowe updated MAPREDUCE-4517:
----------------------------------

    Status: Open  (was: Patch Available)

I agree the messages are too long and printing them every heartbeat is wasteful.  However the messages have proven multiple times to be critical in debugging scheduling issues, and I fear having them turned off completely by default will make debugging issues between the RM and AM very difficult.

I think we could log a lot less and still have the state info we need if we only logged these messages when something in them changes.  Logging the same state every heartbeat isn't useful, but knowing when these states change is.
                
> Too many INFO messages written out during AM to RM heartbeat
> ------------------------------------------------------------
>
>                 Key: MAPREDUCE-4517
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4517
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster
>            Reporter: James Kinley
>            Priority: Minor
>              Labels: patch
>             Fix For: trunk
>
>         Attachments: MAPREDUCE-4517.patch
>
>
> Too many INFO log messages written out during AM to RM heartbeat. Based on default frequency of 1000ms (scheduler.heartbeat.interval-ms) either 2 or 4 INFO messages are written out per second:
> LOG.info("Before Scheduling: " + getStat());
> List<Container> allocatedContainers = getResources();
> LOG.info("After Scheduling: " + getStat());
> if (allocatedContainers.size() > 0) {
>   LOG.info("Before Assign: " + getStat());
>   scheduledRequests.assign(allocatedContainers);
>   LOG.info("After Assign: " + getStat());
> }
> These should probably be changed to DEBUG message to save the log growing too quickly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira