[jira] [Assigned] (HADOOP-18526) Leak of S3AInstrumentation instances via hadoop Metrics references

["Steve Loughran (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-18526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran reassigned HADOOP-18526:
---------------------------------------

    Assignee: Steve Loughran

> Leak of S3AInstrumentation instances via hadoop Metrics references
> ------------------------------------------------------------------
>
>                 Key: HADOOP-18526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18526
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.3.4
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>              Labels: pull-request-available
>
> A heap dump of a process running OOM shows that if a process creates then destroys lots of S3AFS instances, you seem to run out of heap due to references to S3AInstrumentation and the IOStatisticsStore kept via the hadoop metrics registry
> It doesn't look like S3AInstrumentation.close() is being invoked in S3AFS.close(). it should -with the IOStats being snapshotted to a local reference before this happens. This allows for stats of a closed fs to be examined.
> If you look at org.apache.hadoop.ipc.DecayRpcScheduler.MetricsProxy it uses a WeakReference to refer back to the larger object. we should do the same for abfs/s3a bindings. ideally do some template proxy class in hadoop common they can both use.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org