You are viewing a plain text version of this content. The canonical link for it is here.

Posted to yarn-issues@hadoop.apache.org by "Maciej Smole��ski (Jira)" <ji...@apache.org> on 2022/12/16 12:12:00 UTC

[jira] [Created] (YARN-11397) Memory leak when reading aggregated logs from s3 (LogAggregationTFileController::readAggregatedLogs)

Maciej Smole��ski created YARN-11397:
----------------------------------------

             Summary: Memory leak when reading aggregated logs from s3 (LogAggregationTFileController::readAggregatedLogs)
                 Key: YARN-11397
                 URL: https://issues.apache.org/jira/browse/YARN-11397
             Project: Hadoop YARN
          Issue Type: Bug
          Components: log-aggregation
    Affects Versions: 3.2.2
         Environment: Remote logs dir on s3.
            Reporter: Maciej Smole��ski
         Attachments: YarnLogsS3Issue.scala

Reproduction code in the attachment.

When collecting aggregated logs from s3 in a loop (see reproduction code) we can easily see that the number of 'S3AInstrumentation' is increasing although the number of 'S3AFileSystem' is not increasing. It means that 'S3AInstrumentation' is not released together with 'S3AFileSystem' as it should be. The root cause of this seems to be the missing close on S3AFileSystem.

 

Related issues:
https://issues.apache.org/jira/browse/YARN-11039
https://issues.apache.org/jira/browse/HADOOP-18476
The scenario of this bug is different ('thread leak' vs 'memory leak') and 'affected version' are different (3.2.2 in our case), however the root cause might be the same.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org