You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "boaz-gold (via GitHub)" <gi...@apache.org> on 2026/04/06 12:50:00 UTC
[I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
boaz-gold opened a new issue, #15898:
URL: https://github.com/apache/iceberg/issues/15898
### Apache Iceberg version
1.10.0
### Query engine
Spark
### Please describe the bug 🐞
Apache Iceberg version: 1.10.0
Component: org.apache.iceberg.CachingCatalog
Description
CachingCatalog uses a Caffeine cache to hold Table objects. When an entry is evicted (by TTL via cache.expiration-interval-ms or by size via cache.max-total-bytes), the RemovalListener
(MetadataTableInvalidatingRemovalListener) only invalidates related metadata table entries. It does not call table.io().close().
This means any resources held by the FileIO implementation are never released on eviction.
Impact
With io-impl = org.apache.iceberg.aws.s3.S3FileIO:
- Each evicted Table leaves behind a live AWS SDK v2 S3Client
- Each S3Client owns a ScheduledExecutorService (sdk-ScheduledExecutor-N) with background threads for credential refresh (IMDSv2)
- These threads are GC roots — they can never be collected
- In a long-running process (e.g. Spark Thrift Server), threads accumulate without bound until the JVM crashes with os::commit_memory failed; error='Not enough space' (errno=12)
Observed in production (Spark Thrift Server, ~24h uptime):
Total JVM threads: 27,877
sdk-ScheduledExecutor: 27,657
Distinct pool instances: 8,075+
Proof from bytecode
CachingCatalog$MetadataTableInvalidatingRemovalListener.onRemoval() decompiled from iceberg-spark-runtime-3.5_2.12-1.10.0:
// logs debug
// if EXPIRED and not a metadata table: cache.invalidateAll(metadataTableIdentifiers)
// return ← no close() call
There is no table.io().close() call anywhere in the eviction path.
Proposed fix
In CachingCatalog.java, MetadataTableInvalidatingRemovalListener.onRemoval():
if (value != null && value.io() instanceof Closeable) {
try {
((Closeable) value.io()).close();
} catch (IOException e) {
LOG.warn("Failed to close FileIO for evicted table {}", key, e);
}
}
Note: S3FileIO implements Closeable and its close() method calls S3Client.close(), which shuts down the ScheduledExecutorService and releases all threads. This fix is sufficient to resolve the leak.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "boaz-gold (via GitHub)" <gi...@apache.org>.
boaz-gold commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4333563721
@anoopj — you're right, and I apologize for the error. The reference chain I described was backwards.
The direction is S3FileIO → PrefixedS3Client → S3AsyncClient → ScheduledExecutorService → Thread, not the other way around. The AWS SDK has no knowledge of Iceberg's S3FileIO. I should have been more careful before posting.
@xiaoxuandev — thank you for independently reproducing this and for PR #16129. The shared ScheduledExecutorService approach is clean and the NonClosingScheduledExecutorService wrapper is a good detail.
Looking at the code, S3InputFile and S3OutputFile both use the sync S3Client by default (via client()), so the patch covering the sync client builder path appears to address the primary leak path for standard S3FileIO usage. I will be watching the PR closely and happy to test it against our setup once it's ready (not needing to wait for AWS to release)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "boaz-gold (via GitHub)" <gi...@apache.org>.
boaz-gold commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4279356253
@manuzhang - I added production stats to the PR.
Responding on @anoopj two alternatives:
The explicit cleanup method won't help for long-running Thrift servers — there's no point in the lifecycle where a cleanup hook is called between table evictions. The leak still accumulates.
Reference counting inside CachingCatalog sounds like the right approach.
The catalog already owns the cache, so it can map each FileIO instance to a ref count, increment on load, decrement on eviction, and call close() only when the count hits zero.
That correctly handles the shared FileIO case without leaking anything.
wdyt ?
Happy to test any implementation against a live cluster that reproduces the crash deterministically within 24 hours.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4212855804
@boaz-gold Never mind, there is already a PR. Look forward to your future contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "boaz-gold (via GitHub)" <gi...@apache.org>.
boaz-gold commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4302745801
Hi @xiaoxuandev
We're using GlueCatalog wrapped by CachingCatalog.
On FileIOTracker: it closes FileIO instances when catalog.close() is called. In a long-running Spark Thrift Server, catalog.close() is never invoked — the process runs for days until it crashes. So FileIOTracker accumulates every S3Client opened since startup without releasing any of them, regardless of what the TTL cache evicts.
On removing softValues(): the assumption is that a GC-eligible Table will pull the S3Client with it. It won't. Each S3Client owns a ScheduledExecutorService, and live threads are GC roots — they pin the executor, which pins the client. The GC cannot collect them. This is exactly why the crash is os::commit_memory (errno=12) (thread stack exhaustion) and not a heap OOM.
The only thing that actually closes the S3Client is an explicit fileIO.close() call.
The leak is that nothing calls it on cache eviction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "boaz-gold (via GitHub)" <gi...@apache.org>.
boaz-gold commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4212844678
@manuzhang
I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4276426404
The original version of #15910 unconditionally [closed](https://github.com/apache/iceberg/pull/15910/changes/adc2033be7b4612eabd79b4cf0d1cca002ab0c49#r3061553449) the fileIO and would break several existing catalogs that shares the file IO. I recognize your production problem is real, but I think we need to pursue a correct fix that does not break other catalogs.
One way to fix this is to add a method to catalog to clean up an expired table. A catalog implementation knows whether they are reusing fileIO, so they can make the correct decision. Downside is that this is a new API.
Another option is to do some sort of refcount inside `CachingCatalog` - it would be a self contained change, but needs to be designed carefully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "boaz-gold (via GitHub)" <gi...@apache.org>.
boaz-gold commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4325582294
Thanks @anoopj
Here are the results from both diagnostics.
Thrift server uptime at capture: ~3.5 hours — 5,546 leaked sdk-ScheduledExecutor threads already accumulated (~1,580/hour).
---
Forced full GC (jcmd <pid> GC.run)
- sdk-ScheduledExecutor threads before: 5,546
- sdk-ScheduledExecutor threads after (~10s): 5,744
- Total threads before/after: 6,102 → 6,055
Thread count did not decrease — the leaked executor threads are completely unaffected by a full GC.
---
Heap histogram (jmap -histo:live)
Key counts after forced GC:
- S3FileIO — 11,073 instances
- GlueTableOperations — 2,874 instances
- PrefixedS3Client / DefaultS3Client — ~4,439 instances
- Caffeine SoftValueReference — 4,439
The 2,874 GlueTableOperations are the currently cached tables, held by soft references as expected. But there are 11,073 S3FileIO instances alive — ~4x more than active cache
entries. The extra ~8,800 have no surviving parent TableOperations: those were evicted and soft refs cleared. Yet the S3FileIO objects remain strongly reachable via Thread →
PeriodicScheduledTask → S3AsyncClient → S3FileIO. Since live threads are GC roots, full GC cannot collect them regardless of generation.
This rules out the old-gen/weak-reference theory — these objects are not unreachable and awaiting collection, they are anchored by their own threads.
Side observation: 7.1M BaseSnapshot instances in the heap (~1.2GB, ~2,490 per table). This appears to be pushing the cache against its max-total-bytes=2GB limit and increasing
eviction frequency, which worsens the leak rate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4322288262
@boaz-gold I think the crux of the problem is that `FileTracker` is unable to close the `FileIO`. There are two possible root causes for this :
1. Something in the application is holding a strong reference to the `TableOperations`. e.g. Long running queries, or some background thread. You might be able to verify this by doing a heap dump.
2. There are no strong references to `TableOperations`, but the object got into an old gen of the heap, and that region was not collected. You can verify if this is a problem by doing a full GC and see if the SDK threads go down. So you would first do a thread count by doing `jstack <pid> | grep -c "sdk-ScheduledExecutor"`, then do trigger a full GC by doing `jcmd <pid> GC.run` and then run `jstack <pid> | grep -c "sdk-ScheduledExecutor"` again to see if the count is down. Please note that running a full GC will pause the application by a few seconds, so please be careful if you are running on a server facing user traffic.
Once we know the root cause, we can then explore solutions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-6405496061
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] closed issue #15898: CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications
URL: https://github.com/apache/iceberg/issues/15898
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4203476715
@boaz-gold do you plan to submit a fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4316559045
For any potential fix, we need to consider long running queries also. The catalog hands off the table to the engine - if there is a long running query, and we close the file I/O based on a TTL/refcount in caching catalog, the query could break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "xiaoxuandev (via GitHub)" <gi...@apache.org>.
xiaoxuandev commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4292834368
hi @boaz-gold, thanks for reporting the issue.
CachingCatalog doesn't own FileIO and shouldn't manage its lifecycle. GlueCatalog and RESTSessionCatalog already handle this correctly with FileIOTracker at the catalog level — it uses weak keys tied to TableOperations, so FileIO gets closed automatically when the TableOperations is GC'd.
One thing CachingCatalog could do to help: the current softValues() on the Caffeine cache keeps evicted Table objects alive (and their FileIO with them) until memory pressure triggers GC. Switching to weakValues() would let GC reclaim evicted tables promptly, allowing FileIOTracker to close the FileIO much sooner.
Which catalog are you using? That would help identify whether the leak is a missing FileIOTracker in that catalog, or something else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4328155879
Thanks for running the diagnostics, @boaz-gold. One question:
> S3FileIO objects remain strongly reachable via Thread → PeriodicScheduledTask → S3AsyncClient → S3FileIO.
How can `S3AsyncClient` have a reference to `S3FileIO`? S3 client is in the AWS SDK, and should not have any knowledge of Iceberg's `S3FileIO`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "boaz-gold (via GitHub)" <gi...@apache.org>.
boaz-gold commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4321648741
Hi @anoopj, thanks for engaging on this — here's the data you asked for.
JVM / GC config (from our live STG cluster, j-1TCOQHIFPXGDA, EMR 7.12):
- JVM: Corretto 17
- GC: G1GC, -Xmx32G, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent
- GC log enabled: /var/log/spark/driver-gc.log
Observed after ~24h uptime:
- Total JVM threads: 31,041
- sdk-ScheduledExecutor threads (via jstack): 30,830
- Young (Normal) GCs: frequent
- Mixed GCs completed: 1,202
- Full GCs: 0
The old gen is being scanned regularly via mixed GCs, but the sdk-ScheduledExecutor threads persist through all of them. My understanding is that these are live Java threads — they appear as running/waiting in jstack — and live threads are GC roots, so weak-reference cleanup wouldn't reclaim them regardless of GC frequency. But I might be missing something about how FileIOTracker interacts with GlueCatalog + CachingCatalog specifically — if you see a gap in that reasoning I'd genuinely appreciate the correction.
Regarding your point about long-running queries: that's a valid concern and I don't have a clean answer. In my case, with a 30-second TTL, a table can be evicted while a query that loaded it is still reading — calling fileIO.close() at that point could break in-flight reads. also I noticed that the test in PR #15910 still asserts verify(mockIO).close(), which seemed to conflict with the removal of the close() call — not sure if that's intentional or an oversight.
Of the two directions you suggested, the catalog-level cleanup hook feels more tractable to us since each catalog implementation knows its own FileIO ownership semantics. The reference counting approach seems cleaner in principle but I am not sure how the decrement side would work without changing the loadTable contract. That said, I am very much in learning mode here and would follow whatever direction the maintainers think is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "xiaoxuandev (via GitHub)" <gi...@apache.org>.
xiaoxuandev commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4328230200
@boaz-gold Thanks for sharing the additional diagnostics, the heap histogram and full GC results are helpful and confirm the root cause.
### Reproduction
I am able to reproduce the thread leak on EMR 7.12 using Spark Thrift Server with cache.expiration-interval-ms=10 and io-impl=S3FileIO. We observed sdk-ScheduledExecutor threads growing monotonically (947 → 1067 in ~5 minutes), matching your production pattern.
### Root Cause
Your analysis is correct. The reference chain is:
sdk-ScheduledExecutor thread (GC root)
→ ScheduledExecutorService
→ S3Client internals
→ S3FileIO
Each time CachingCatalog evicts a table entry (via TTL or max-total-bytes pressure) and GlueCatalog reloads it, newTableOps() creates a new S3FileIO → new S3Client → new ScheduledExecutorService with its own sdk-ScheduledExecutor threads. The old S3Client is never closed, and since its threads are live (GC roots), the entire chain is strongly reachable. Full GC cannot help — these are not unreachable objects awaiting collection, they are anchored by their own threads.
This issue manifests specifically in long-running JVM processes (Thrift Server, Structured Streaming, long-lived notebook sessions) where threads accumulate over hours/days. Short-lived spark-submit jobs are unaffected because the JVM exits before threads accumulate to problematic levels.
### Fix
Rather than closing FileIO on eviction (which could break long-running queries still using the evicted table's S3Client as
@anoopj pointed out), we share a single ScheduledExecutorService (pool size 5, matching the AWS SDK v2 default) across all S3Clients created by Iceberg's AWS client factories. This is done via ClientOverrideConfiguration.scheduledExecutorService().
With this fix, no matter how many S3Clients are created or evicted, the total sdk-ScheduledExecutor thread count stays fixed at 5. We verified on EMR 7.12 Thrift Server: after the fix, sdk-ScheduledExecutor count is 0 and iceberg-shared-sdk-executor count is stable at 5 (previously 718 and keep growing).
Please take a look at the fix here: https://github.com/apache/iceberg/pull/16129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4315131745
@boaz-gold What JVM version/garbage collector do you use? Also, is this a busy server where there are many minor collections? (You can verify this by enabling the GC logs/metrics)
One theory I have is that the `CachingCatalog` evicts the table after the TTL (30 s), but the table/ops objects gets promoted to the old gen of the JVM heap. So fileIO tracker is not able to clean up because it is a weak reference. Old gen weak references get cleared whenever a GC cycle marks the old generation and finds the referent unreachable. In G1, that's mixed GCs and full GCs. They are NOT cleared by young-only collections.
So if you have a large JVM heaps where old gen collections never run, the fileIO can leak.
cc @nastra for his thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "boaz-gold (via GitHub)" <gi...@apache.org>.
boaz-gold commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4275550650
I want to flag that PR https://github.com/apache/iceberg/pull/15910 no longer fixes this issue.
The original commit had the right idea — it added table.io().close() in the RemovalListener. But after reviewer feedback about shared FileIO instances, the close call was removed entirely. What's left in the PR is just changing RemovalCause.EXPIRED.equals(cause) to cause.wasEvicted(), which has nothing to do with the thread leak.
I'm hitting this in production. After 24h with ~300 Iceberg tables and a 30s cache TTL we end up with ~3,500 live S3Client instances and ~28,000 leaked sdk-ScheduledExecutor threads, crashing with os::commit_memory failed; error='Not enough space' (errno=12) — thread stack exhaustion, not heap OOM.
Had to roll back to EMR 7.9 as a workaround.
On the shared FileIO concern — I think the right answer is to still call close() and let each FileIO implementation decide what that means. S3FileIO creates a client per table, it's never shared, so closing on eviction is always safe. If some catalog shares a single FileIO across tables, that implementation should make close() a no-op, not the other
way around.
Would be happy to test any proposed fix on a live cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-4275556631
@boaz-gold Thanks for reporting. Can you share your findings in #15910?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications [iceberg]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on issue #15898:
URL: https://github.com/apache/iceberg/issues/15898#issuecomment-6755073141
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org