[jira] [Commented] (HADOOP-18526) Leak of S3AInstrumentation instances via hadoop Metrics references

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-18526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17641839#comment-17641839 ] 

ASF GitHub Bot commented on HADOOP-18526:
-----------------------------------------

steveloughran commented on PR #5144:
URL: https://github.com/apache/hadoop/pull/5144#issuecomment-1333603252

   tested aws london w/ ` -Dparallel-tests -DtestsThreadCount=10 -Dmarkers=delete`
   
   s3a fs was set to log at TRACE so print stack traces on all fs creation...just search for "creation stack" in the output to look. no problems when this tracing is enabled.
   
   saw HADOOP-18351 again, so have just rebased for future runs. 
   
   think this is ready for review now 
   
   + @mukund-thakur @mehakmeet @dannycjones @ashutoshcipher @HarshitGupta11 




> Leak of S3AInstrumentation instances via hadoop Metrics references
> ------------------------------------------------------------------
>
>                 Key: HADOOP-18526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18526
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.3.4
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>              Labels: pull-request-available
>
> A heap dump of a process running OOM shows that if a process creates then destroys lots of S3AFS instances, you seem to run out of heap due to references to S3AInstrumentation and the IOStatisticsStore kept via the hadoop metrics registry
> It doesn't look like S3AInstrumentation.close() is being invoked in S3AFS.close(). it should -with the IOStats being snapshotted to a local reference before this happens. This allows for stats of a closed fs to be examined.
> If you look at org.apache.hadoop.ipc.DecayRpcScheduler.MetricsProxy it uses a WeakReference to refer back to the larger object. we should do the same for abfs/s3a bindings. ideally do some template proxy class in hadoop common they can both use.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org