[jira] [Updated] (SPARK-58438) HadoopDelegationTokenManager leaks FileSystem cache entries on every token renewal

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-58438:
-----------------------------------
    Labels: pull-request-available  (was: )

> HadoopDelegationTokenManager leaks FileSystem cache entries on every token renewal
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-58438
>                 URL: https://issues.apache.org/jira/browse/SPARK-58438
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 4.2.0
>            Reporter: Parth Chandra
>            Priority: Major
>              Labels: pull-request-available
>
> {{{}HadoopDelegationTokenManager{}}}'s periodic token-renewal path creates a fresh
> {{UserGroupInformation}} (UGI) on every renewal cycle but never releases the
> {{FileSystem}} instances cached for that UGI, so entries accumulate in the static
> {{FileSystem.CACHE}} for the lifetime of the driver JVM.
> h4. Details
> On each renewal cycle {{updateTokensTask()}} calls {{{}doLogin(){}}}, which returns a
> *new* UGI in the two credential modes:
>  * keytab: {{UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)}}
>  * ccache: {{UserGroupInformation.getUGIFromTicketCache(ccache, user)}}
> Providers invoked inside {{freshUGI.doAs
> { ... }
> }} call {{{}FileSystem.get(...){}}}, which
> caches {{FileSystem}} instances in {{FileSystem.CACHE}} keyed (in part) by the UGI.
> Cache entries are only evicted by {{FileSystem.close}} / {{{}FileSystem.closeAllForUGI{}}}.
> The renewal path never calls either, so each cycle leaves behind {{FileSystem}}
> instances for a throwaway UGI that is never used again.
> The one-shot path {{obtainDelegationTokens(creds)}} already cleans up correctly:
> {code:scala}
>   if (!currentUser.equals(freshUGI)) {
>     FileSystem.closeAllForUGI(freshUGI)
>   }
>   {code}
> but the equivalent cleanup is missing from {{obtainTokensAndScheduleRenewal}} /
> {{{}updateTokensTask{}}}.
> h4. Impact
> Long-running applications that renew delegation tokens (keytab, or ccache-based
> renewal) accumulate one UGI's worth of cached {{FileSystem}} handles per renewal
> interval. With default settings the renewal interval is on the order of hours, so the
> growth is slow, but it is unbounded and can lead to gradually increasing memory and
> open file/connection usage (and eventually resource exhaustion) over the lifetime of a
> driver that runs for days or weeks.
> The proxy-user branch of {{doLogin()}} returns the shared {{getCurrentUser()}} and must
> NOT be closed; the guard {{!currentUser.equals(freshUGI)}} correctly excludes it.
> h4. Suggested fix
> Mirror the one-shot path: after the renewal {{doAs}} completes, close the fresh UGI's
> cached filesystems, guarded by {{!currentUser.equals(freshUGI)}} so the shared
> current/proxy user is never closed.
> Notes
> Discovered during review of SPARK-38954 (PR #57285); it is a pre-existing issue in the
> Kerberos renewal path and is being tracked separately per reviewer request so that
> SPARK-38954 keeps the Kerberos paths unchanged.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org