You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dhruvarya-db (via GitHub)" <gi...@apache.org> on 2026/06/23 01:01:54 UTC
[PR] fix: avoid lost-wakeup hang in DeleteFileIndex::get_deletes_for_data_file [iceberg-rust]
dhruvarya-db opened a new pull request, #2696:
URL: https://github.com/apache/iceberg-rust/pull/2696
## Which issue does this PR close?
N/A — concurrency bug found while reviewing #2590.
## What changes are included in this PR?
Fixes a lost-wakeup hang in `DeleteFileIndex::get_deletes_for_data_file`.
## The bug
`DeleteFileIndex` populates asynchronously on a background task that signals completion with `tokio::sync::Notify::notify_waiters()`. Per the tokio docs, `notify_waiters()` stores no permit and only wakes `Notified` futures that have already been **created**.
`get_deletes_for_data_file` observes the `Populating` state, **releases the read lock**, and only *then* creates the `Notified` future. If the populating task flips the state to `Populated` and calls `notify_waiters()` in that window, the wakeup is lost and the query awaits forever.
The window is normally only a few instructions wide and is effectively never hit, because the querier usually registers before the populator is even scheduled. This is reachable from the `SnapshotValidator` commit path (apache/iceberg-rust#2590), which drops the channel sender immediately before querying — making the populator finish right as the query starts.
## The fix
Create the `Notified` future (via `notified_owned`) **while still holding the read lock**. The populating task cannot call `notify_waiters()` until it acquires the write lock, which is blocked while the read lock is held — so the notification is guaranteed to happen *after* the future has been created, and tokio guarantees such notifications are delivered. No re-check needed.
The same latent pattern exists in `arrow/delete_filter.rs` and `arrow/caching_delete_file_loader.rs`; this PR fixes only the index site, and I'm happy to follow up on the others.
## Are these changes tested?
The change is verified by a deterministic reproduction that uses a small test-only seam to make the otherwise nanosecond-wide window reliable. To keep this production change minimal, the seam + test live in companion PRs on my fork rather than here:
- Reproduction only — the test **fails** (hangs → timeout), demonstrating the bug: https://github.com/dhruvarya-db/iceberg-rust/pull/9
- The same fix **with** the regression test — the test **passes**: https://github.com/dhruvarya-db/iceberg-rust/pull/10
I'm happy to fold a regression test directly into this PR if maintainers prefer.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: avoid lost-wakeup hang in DeleteFileIndex::get_deletes_for_data_file [iceberg-rust]
Posted by "dhruvarya-db (via GitHub)" <gi...@apache.org>.
dhruvarya-db commented on PR #2696:
URL: https://github.com/apache/iceberg-rust/pull/2696#issuecomment-4793331417
@CTTY Added a test for the fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: avoid lost-wakeup hang in DeleteFileIndex::get_deletes_for_data_file [iceberg-rust]
Posted by "dhruvarya-db (via GitHub)" <gi...@apache.org>.
dhruvarya-db commented on PR #2696:
URL: https://github.com/apache/iceberg-rust/pull/2696#issuecomment-4784121406
@CTTY Would this [kind of a test ](https://github.com/dhruvarya-db/iceberg-rust/pull/10/changes) be okay? It adds some test-only logic to the main code which is why I was avoiding it:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: avoid lost-wakeup hang in DeleteFileIndex::get_deletes_for_data_file [iceberg-rust]
Posted by "dhruvarya-db (via GitHub)" <gi...@apache.org>.
dhruvarya-db commented on PR #2696:
URL: https://github.com/apache/iceberg-rust/pull/2696#issuecomment-4794931758
@CTTY I have removed the test. Could you take another look when you get a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: avoid lost-wakeup hang in DeleteFileIndex::get_deletes_for_data_file [iceberg-rust]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #2696:
URL: https://github.com/apache/iceberg-rust/pull/2696#discussion_r3480650837
##########
crates/iceberg/src/delete_file_index.rs:
##########
@@ -86,17 +86,21 @@ impl DeleteFileIndex {
data_file: &DataFile,
seq_num: Option<i64>,
) -> Vec<FileScanTaskDeleteFile> {
- let notifier = {
+ // Create the `Notified` while holding the read lock. `notify_waiters()` stores no
Review Comment:
I think the comment describes the wrong mechanism, and since it's the only explanation of why this is correct I'd want it tightened before merge.
The populator drops the write lock *before* it calls `notify_waiters()` (over in the populate task), so holding this read lock doesn't actually block the signal — a `notify_waiters()` can still fire the instant we drop the guard, before we first poll the future. What saves us is narrower: tokio captures the `notify_waiters_calls` counter when the `Notified` is constructed, so a `notify_waiters()` that lands after construction bumps the counter and the future resolves on the first poll. The read lock's real job is just to stop the state flipping to `Populated` between the read and the construction.
Worth getting right because someone porting this to the other two sites will reason from this comment — and the same wording would steer them toward a borrowed `Notified<'_>` or a `notify_one` context where the logic doesn't hold. Could we reword it around the counter-captured-at-construction guarantee?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: avoid lost-wakeup hang in DeleteFileIndex::get_deletes_for_data_file [iceberg-rust]
Posted by "dhruvarya-db (via GitHub)" <gi...@apache.org>.
dhruvarya-db commented on code in PR #2696:
URL: https://github.com/apache/iceberg-rust/pull/2696#discussion_r3483534225
##########
crates/iceberg/src/delete_file_index.rs:
##########
@@ -86,17 +86,21 @@ impl DeleteFileIndex {
data_file: &DataFile,
seq_num: Option<i64>,
) -> Vec<FileScanTaskDeleteFile> {
- let notifier = {
+ // Create the `Notified` while holding the read lock. `notify_waiters()` stores no
Review Comment:
> The populator drops the write lock before it calls notify_waiters() (over in the populate task), so holding this read lock doesn't actually block the signal — a notify_waiters() can still fire the instant we drop the guard, before we first poll the future.
```
{
let mut guard = state.write().unwrap();
*guard = DeleteFileIndexState::Populated(populated_delete_file_index);
}
```
If the write lock has been taken and dropped already before we acquire this read lock, the state would have already been set to `DeleteFileIndexState::Populated`. We only need the notifier for the case where the state is `DeleteFileIndexState::Populating`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: avoid lost-wakeup hang in DeleteFileIndex::get_deletes_for_data_file [iceberg-rust]
Posted by "dhruvarya-db (via GitHub)" <gi...@apache.org>.
dhruvarya-db commented on code in PR #2696:
URL: https://github.com/apache/iceberg-rust/pull/2696#discussion_r3483539638
##########
crates/iceberg/src/delete_file_index.rs:
##########
@@ -86,17 +86,21 @@ impl DeleteFileIndex {
data_file: &DataFile,
seq_num: Option<i64>,
) -> Vec<FileScanTaskDeleteFile> {
- let notifier = {
+ // Create the `Notified` while holding the read lock. `notify_waiters()` stores no
Review Comment:
Read the comment again, the suggestion makes sense. Will tighten the comment
##########
crates/iceberg/src/delete_file_index.rs:
##########
@@ -86,17 +86,21 @@ impl DeleteFileIndex {
data_file: &DataFile,
seq_num: Option<i64>,
) -> Vec<FileScanTaskDeleteFile> {
- let notifier = {
+ // Create the `Notified` while holding the read lock. `notify_waiters()` stores no
Review Comment:
Read the suggestion again, it makes sense. Will tighten the code comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: avoid lost-wakeup hang in DeleteFileIndex::get_deletes_for_data_file [iceberg-rust]
Posted by "dhruvarya-db (via GitHub)" <gi...@apache.org>.
dhruvarya-db commented on code in PR #2696:
URL: https://github.com/apache/iceberg-rust/pull/2696#discussion_r3483802946
##########
crates/iceberg/src/delete_file_index.rs:
##########
@@ -86,17 +86,21 @@ impl DeleteFileIndex {
data_file: &DataFile,
seq_num: Option<i64>,
) -> Vec<FileScanTaskDeleteFile> {
- let notifier = {
+ // Create the `Notified` while holding the read lock. `notify_waiters()` stores no
Review Comment:
Updated the comment @laskoviymishka . Does it make sense now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: avoid lost-wakeup hang in DeleteFileIndex::get_deletes_for_data_file [iceberg-rust]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka merged PR #2696:
URL: https://github.com/apache/iceberg-rust/pull/2696
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org