You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Maciej Smolenski (Jira)" <ji...@apache.org> on 2022/12/20 10:46:00 UTC
[jira] [Updated] (SPARK-41599) Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher
[ https://issues.apache.org/jira/browse/SPARK-41599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maciej Smolenski updated SPARK-41599:
-------------------------------------
Attachment: InProcLaunchFsIssue.scala
> Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-41599
> URL: https://issues.apache.org/jira/browse/SPARK-41599
> Project: Spark
> Issue Type: Bug
> Components: Deploy, YARN
> Affects Versions: 3.1.2
> Reporter: Maciej Smolenski
> Priority: Major
> Attachments: InProcLaunchFsIssue.scala
>
>
> When submitting spark application in kerberos environment the credentials of 'current user' (UserGroupInformation.getCurrentUser()) are being modified.
> Filesystem.CACHE entries contain 'current user' (with user credentials) as a key.
> Submitting many spark applications using InProcessLauncher cause that FileSystem.CACHE becomes bigger and bigger.
> Finally process exits because of OutOfMemory error.
> Code for reproduction attached.
>
> Output from running 'jmap -histo' on reproduction jvm shows that the number of FileSystem$Cache$Key increases in time:
> time: numInstances class
> 1671529324: 9 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671529384: 18 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671529445: 27 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671529505: 36 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671529565: 45 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671529625: 54 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671529686: 63 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671529746: 72 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671529806: 81 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671529866: 90 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671529927: 99 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671529987: 108 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530047: 117 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530108: 126 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530168: 135 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530228: 144 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530288: 153 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530349: 162 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530409: 170 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530469: 179 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530530: 188 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530590: 197 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530650: 206 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530710: 219 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530771: 226 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530831: 231 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530891: 240 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671530952: 249 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671531012: 258 org.apache.hadoop.fs.FileSystem$Cache$Key
> 1671531072: 270 org.apache.hadoop.fs.FileSystem$Cache$Key
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org