You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Maciej Smole��ski (Jira)" <ji...@apache.org> on 2022/12/16 12:14:00 UTC
[jira] [Updated] (YARN-11397) Memory leak when reading aggregated logs from s3 (LogAggregationTFileController::readAggregatedLogs)
[ https://issues.apache.org/jira/browse/YARN-11397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maciej Smole��ski updated YARN-11397:
-------------------------------------
Description:
Reproduction code in the attachment.
When collecting aggregated logs from s3 in a loop (see reproduction code) we can easily see that the number of 'S3AInstrumentation' is increasing although the number of 'S3AFileSystem' is not increasing. It means that 'S3AInstrumentation' is not released together with 'S3AFileSystem' as it should be. The root cause of this seems to be the missing close on S3AFileSystem.
was:
Reproduction code in the attachment.
When collecting aggregated logs from s3 in a loop (see reproduction code) we can easily see that the number of 'S3AInstrumentation' is increasing although the number of 'S3AFileSystem' is not increasing. It means that 'S3AInstrumentation' is not released together with 'S3AFileSystem' as it should be. The root cause of this seems to be the missing close on S3AFileSystem.
Related issues:
https://issues.apache.org/jira/browse/YARN-11039
https://issues.apache.org/jira/browse/HADOOP-18476
The scenario of this bug is different ('thread leak' vs 'memory leak') and 'affected version' are different (3.2.2 in our case), however the root cause might be the same.
> Memory leak when reading aggregated logs from s3 (LogAggregationTFileController::readAggregatedLogs)
> ----------------------------------------------------------------------------------------------------
>
> Key: YARN-11397
> URL: https://issues.apache.org/jira/browse/YARN-11397
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
> Affects Versions: 3.2.2
> Environment: Remote logs dir on s3.
> Reporter: Maciej Smole��ski
> Priority: Critical
> Attachments: YarnLogsS3Issue.scala
>
>
> Reproduction code in the attachment.
> When collecting aggregated logs from s3 in a loop (see reproduction code) we can easily see that the number of 'S3AInstrumentation' is increasing although the number of 'S3AFileSystem' is not increasing. It means that 'S3AInstrumentation' is not released together with 'S3AFileSystem' as it should be. The root cause of this seems to be the missing close on S3AFileSystem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org