You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Kingsley-working (via GitHub)" <gi...@apache.org> on 2026/10/05 04:52:14 UTC
[I] Core: AuthSessionCache's exiting evictor pool registers a never-removed JVM shutdown hook per OAuth2Manager, pinning the creating thread's inheritable thread-locals (leaks closed Spark sessions) [iceberg]
Kingsley-working opened a new issue, #18377:
URL: https://github.com/apache/iceberg/issues/18377
### Apache Iceberg version
1.11.0 (also present on `main` today: `core/src/main/java/org/apache/iceberg/rest/auth/AuthSessionCache.java` L54-58, `OAuth2Manager.java` L216-217, `util/ThreadPools.java` L101-104)
### Query engine
Spark 4.1.1, as a long-lived Spark Connect server (many short-lived sessions on one driver JVM), REST catalog with the default `oauth2` auth (Apache Polaris).
### Please describe the bug 🐞
`OAuth2Manager.newSessionCache` creates its `AuthSessionCache` through the public constructor, which builds a dedicated single-thread pool with `ThreadPools.newExitingWorkerPool(name + "-auth-session-evict", 1)`. That goes through Guava's `MoreExecutors.getExitingExecutorService`, which registers a JVM shutdown hook (`Runtime.addShutdownHook(new Thread(...))`) that is never removed. `AuthSessionCache.close()` shuts the executor down but the hook stays registered.
On a long-lived JVM that creates many catalog instances (Spark creates one catalog plugin, hence one `OAuth2Manager`, per Spark session) this has two effects:
1. `java.lang.ApplicationShutdownHooks.hooks` grows by one unstarted `Thread` per catalog instance for the life of the JVM.
2. Each hook `Thread` is constructed on the calling thread, so it copies that thread's inheritable thread-locals. Spark's `SparkSession.activeThreadSession` is an `InheritableThreadLocal`, so every hook pins the Spark session that created the catalog, including sessions that have since been closed, together with their catalogs, table caches and every loaded `TableMetadata` (full snapshot history). Heap dump, Eclipse MAT, GC-root path with weak/soft references excluded (it is the only path):
```
java.lang.ApplicationShutdownHooks.hooks
└ Thread "DelayedShutdownHook-for-java.util.concurrent.ThreadPoolExecutor@…" (never started)
└ inheritableThreadLocals → ThreadLocalMap → Entry.value
└ org.apache.spark.sql.classic.SparkSession (a CLOSED Spark Connect session, ~36 MB retained)
```
Measured on a shared Spark Connect driver (4 GiB heap): 319 such hook threads after ~3 h; `ApplicationShutdownHooks` retained 51% of the live heap; releasing or idle-expiring sessions returned 0-9% of it; the heap left after each full GC only grew until the driver ran out of memory, roughly daily.
The same pattern (a per-call exiting pool, hence a hook per call) exists in `RewriteDataFilesSparkAction.rewriteService()` (`Rewrite-Service`: 296 hooks after one `rewrite_data_files` sweep), `RewritePositionDeleteFilesSparkAction.rewriteService()` and `BaseProcedure.executorService(...)` when a pool size is passed.
#15031 covers the hooks of the *static* pools in `ThreadPools` (class-loader leaks on hot reload). This report is about the per-instance and per-call pools, whose hooks multiply with the number of catalogs and calls.
**To reproduce:** on one JVM, create N `OAuth2Manager` instances (for example N Spark Connect sessions that each read from a REST catalog with OAuth2), close them, force a full GC, then look at `java.lang.ApplicationShutdownHooks.hooks` (N new entries) or a live heap histogram (the N closed `SparkSession` objects are still reachable).
**Suggested fix:** per-instance and per-call caches should not use an "exiting" pool. Options:
- (a) one shared static daemon evictor for all `AuthSessionCache` instances, passed as a plain `Executor` so `close()` does not shut it down, with its thread created with inheritable thread-locals disabled (`new Thread(null, r, name, 0, false)`) so even that single thread pins nothing;
- (b) a plain daemon pool that `close()` shuts down, with no shutdown hook;
- (c) if a hook is kept, remove it in `close()` and construct the hook thread with inherited thread-locals disabled.
We run (a) in production as an `OAuth2Manager` subclass loaded through `rest.auth.type=<fqcn>` (overriding `newSessionCache` and using the package-private `AuthSessionCache(Duration, Executor, Ticker)` constructor): closed sessions are collected at the next GC and the heap floor returns.
### Willingness to contribute
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core: AuthSessionCache's exiting evictor pool registers a never-removed JVM shutdown hook per OAuth2Manager, pinning the creating thread's inheritable thread-locals (leaks closed Spark sessions) [iceberg]
Posted by "Kingsley-working (via GitHub)" <gi...@apache.org>.
Kingsley-working commented on issue #18377:
URL: https://github.com/apache/iceberg/issues/18377#issuecomment-5992284197
Thanks for picking this up. As the reporter I ticked "willing to contribute": we have run option (a) in production since 2026-10-02 on a shared Spark Connect driver, and I'd like to upstream it with the Gradle build and the hook-count regression test from that deployment. #18378 currently fails `:iceberg-core:compileJava` (errorprone `UnnecessaryMethodReference`) and is marked unreviewed. I'll open a PR shortly, and maintainers can pick whichever they prefer. Neither PR covers the per-call pools in the Spark rewrite actions and `BaseProcedure`; I can take those as a follow-up.
cc @nastra @danielcweeks @adutra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core: AuthSessionCache's exiting evictor pool registers a never-removed JVM shutdown hook per OAuth2Manager, pinning the creating thread's inheritable thread-locals (leaks closed Spark sessions) [iceberg]
Posted by "developer-rpai (via GitHub)" <gi...@apache.org>.
developer-rpai commented on issue #18377:
URL: https://github.com/apache/iceberg/issues/18377#issuecomment-5998271967
Hi Kingsley — thanks for sharing the context on your production deployment. I am a data architect with 15+ years in data engineering, and I am looking to get more involved in the Iceberg community beyond one-off fixes. Happy for maintainers to pick whichever PR fits best here. Is there a good place to plug in — e.g. the Spark 4.2 tracking items or auth-related follow-ups? I would love to contribute more substantially.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core: AuthSessionCache's exiting evictor pool registers a never-removed JVM shutdown hook per OAuth2Manager, pinning the creating thread's inheritable thread-locals (leaks closed Spark sessions) [iceberg]
Posted by "developer-rpai (via GitHub)" <gi...@apache.org>.
developer-rpai commented on issue #18377:
URL: https://github.com/apache/iceberg/issues/18377#issuecomment-5989809111
I would like to work on this. I will open a PR with a fix plus regression tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org