You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org> on 2013/10/15 23:49:47 UTC

[jira] [Updated] (YARN-202) Log Aggregation generates a storm of fsync() for namenode

     [ https://issues.apache.org/jira/browse/YARN-202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod Kumar Vavilapalli updated YARN-202:
-----------------------------------------

    Issue Type: Sub-task  (was: Bug)
        Parent: YARN-431

> Log Aggregation generates a storm of fsync() for namenode
> ---------------------------------------------------------
>
>                 Key: YARN-202
>                 URL: https://issues.apache.org/jira/browse/YARN-202
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 2.0.2-alpha, 0.23.4
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>             Fix For: 3.0.0, 2.0.3-alpha, 0.23.5
>
>         Attachments: yarn-202.patch
>
>
> When the log aggregation is on, write to each aggregated container log causes hflush() to be called. For large clusters, this can creates a lot of fsync() calls for namenode. 
> We have seen 6-7x increase in the average number of fsync operations compared to 1.0.x on a large busy cluster. Over 99% of fsync ops were for log aggregation writing to tmp files.



--
This message was sent by Atlassian JIRA
(v6.1#6144)