You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@tez.apache.org by "Bikas Saha (JIRA)" <ji...@apache.org> on 2015/09/12 06:00:47 UTC

[jira] [Comment Edited] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

    [ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741890#comment-14741890 ] 

Bikas Saha edited comment on TEZ-2774 at 9/12/15 4:00 AM:
----------------------------------------------------------

Attaching v4 patch with some modification. Added and removed logs. Since its all in the AM, just experimented using MockApp code on dev box for the AM log. This does not account for scheduler logging. Will try running a real job later to check on that. The simulation AM log was 1.5MB in size for a 200x200 scatter-gather job.

Biggest removal is counter logging in attempt finished event since it duplicates counter logging in task finished event. That should save a lot of space and make up for some of the logs that I added back. Counter logging removal does not reduce simulation logs since they have not counters.
Looking ahead we could stop logging counters completely in the history logs and that should save a ton of space.


was (Author: bikassaha):
Attaching v4 patch with some modification. Added and removed logs. Since its all in the AM, just experimented using MockApp code on dev box for the AM log. This does not account for scheduler logging. Will try running a real job later to check on that. The simulation AM log was 1.5MB in size.

Biggest removal is counter logging in attempt finished event since it duplicates counter logging in task finished event. That should save a lot of space and make up for some of the logs that I added back. 
Looking ahead we could stop logging counters completely in the history logs and that should save a ton of space.

> Reduce logging in the AM, and parts of the runtime
> --------------------------------------------------
>
>                 Key: TEZ-2774
>                 URL: https://issues.apache.org/jira/browse/TEZ-2774
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>         Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, TEZ-2774.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)