You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2018/10/18 10:21:00 UTC

[jira] [Created] (SPARK-25766) AMCredentialRenewer can leak FS clients

Steve Loughran created SPARK-25766:
--------------------------------------

             Summary: AMCredentialRenewer can leak FS clients
                 Key: SPARK-25766
                 URL: https://issues.apache.org/jira/browse/SPARK-25766
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 2.4.0
            Reporter: Steve Loughran


AMCredentialRenewer's scheduled {{writeNewCredentialsToHDFS}} operation creates a new FS connector each time, so as to access the store with refreshed credentials

but it doesn't close it after, so any resources used by the client are kept around. This is more expensive with the cloud store connectors which create thread pools. 

It should call {{remoteFs.close()}} at the end of its work.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org