You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dhruvarya-db (via GitHub)" <gi...@apache.org> on 2026/07/21 21:24:53 UTC

[PR] fix: avoid lost-wakeup hang in get_equality_delete_predicate_for_delete_file_path [iceberg-rust]

dhruvarya-db opened a new pull request, #2873:
URL: https://github.com/apache/iceberg-rust/pull/2873

   ## Which issue does this PR close?
   
   <!-- No issue; a latent concurrency bug found while reviewing the delete-file loading paths. -->
   
   N/A — a latent lost-wakeup bug in the equality-delete loading path.
   
   ## What changes are included in this PR?
   
   Fixes a lost-wakeup hang in `DeleteFilter::get_equality_delete_predicate_for_delete_file_path` (`crates/iceberg/src/arrow/delete_filter.rs`).
   
   `tokio::sync::Notify::notify_waiters()` stores no permit and only wakes `Notified` futures that already exist. Previously the consumer observed the `Loading` state, released the read lock, and only then created its `Notified`. The losing interleaving:
   
   1. The entry is `EqDelState::Loading(notify)`.
   2. The consumer sees `Loading`, clones the `Arc<Notify>`, and **releases the read lock**.
   3. The loader (`insert_equality_delete`) transitions the entry to `Loaded` and calls `notify_waiters()`. No `Notified` exists yet, so the wakeup is dropped.
   4. The consumer calls `notify.notified().await`, creating its `Notified` **after** step 3 — it waits for the next `notify_waiters()`, which never comes → hangs forever.
   
   **Fix:** create the `Notified` (via `notified_owned`) while still holding the read lock. The loader cannot call `notify_waiters()` until it takes the write lock, which is blocked while the read lock is held, so the notification is guaranteed to happen after the future is created and is delivered.
   
   This is latent today (the loader does real work before signaling, so the window is rarely hit), but is one scheduling hiccup from a permanent hang. It is the same bug class as #2696 (`DeleteFileIndex`) and #2859 (positional deletes).
   
   ## Are these changes tested?
   
   A deterministic regression test is possible, but it requires a small `#[cfg(test)]` timing seam in the consumer: unlike the positional-delete path (#2859), this method does not hand the notifier back to its caller, so the losing interleaving cannot be forced through the public API alone — the test must pause the consumer in the window between observing `Loading` and registering its `Notified`. To keep this production change minimal and seam-free, the seam + tests live in companion PRs on my fork:
   
   - Reproduction only — the test **fails** (hangs → timeout), demonstrating the bug: https://github.com/dhruvarya-db/iceberg-rust/pull/17
   - The same fix **with** the regression test (using the seam) — the test **passes**: https://github.com/dhruvarya-db/iceberg-rust/pull/14
   
   I'm happy to fold the seam + regression test into this PR if maintainers prefer that over keeping the production change seam-free.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: avoid lost-wakeup hang in get_equality_delete_predicate_for_delete_file_path [iceberg-rust]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2873:
URL: https://github.com/apache/iceberg-rust/pull/2873#issuecomment-5403185258

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: avoid lost-wakeup hang in get_equality_delete_predicate_for_delete_file_path [iceberg-rust]

Posted by "dhruvarya-db (via GitHub)" <gi...@apache.org>.
dhruvarya-db commented on PR #2873:
URL: https://github.com/apache/iceberg-rust/pull/2873#issuecomment-5515726819

   @dannycjones Thanks for flagging this! I somehow thought that this was merged. I will revive this shortly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: avoid lost-wakeup hang in get_equality_delete_predicate_for_delete_file_path [iceberg-rust]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2873: fix: avoid lost-wakeup hang in get_equality_delete_predicate_for_delete_file_path
URL: https://github.com/apache/iceberg-rust/pull/2873


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: avoid lost-wakeup hang in get_equality_delete_predicate_for_delete_file_path [iceberg-rust]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2873:
URL: https://github.com/apache/iceberg-rust/pull/2873#issuecomment-5486712791

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: avoid lost-wakeup hang in get_equality_delete_predicate_for_delete_file_path [iceberg-rust]

Posted by "dannycjones (via GitHub)" <gi...@apache.org>.
dannycjones commented on PR #2873:
URL: https://github.com/apache/iceberg-rust/pull/2873#issuecomment-5509773035

   @dhruvarya-db I think this change is important to get merged.
   
   Would you like to continue owning this PR, or would you prefer someone take ownership in a new PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: avoid lost-wakeup hang in get_equality_delete_predicate_for_delete_file_path [iceberg-rust]

Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #2873:
URL: https://github.com/apache/iceberg-rust/pull/2873#discussion_r3629675600


##########
crates/iceberg/src/arrow/delete_filter.rs:
##########


Review Comment:
   Also line 112 (`try_start_eq_del_load`) above. The loader claims the entry with `try_start_eq_del_load`, which inserts `EqDelState::Loading(notify_A)` and returns `notify_A`. The caller drops `notify_A` unused and calls `insert_equality_delete`, which inserts a second `EqDelState::Loading(notify_B)` over the same key and only ever calls `notify_waiters()` on `notify_B`. A reader that observed the entry while it held `notify_A` waits on a notifier that is never signalled.
   
   This is the same class of bug this PR fixes, and it survives the fix. It was flagged on #2859: https://github.com/apache/iceberg-rust/pull/2859#discussion_r3629462979
   
   The window is small today (the two calls run back to back with no `.await` between them, and readers run after the load phase), so this is latent rather than active. Since it is the same root cause and the same file, closing it here rather than in a follow-up keeps the delete-loading paths consistent. `insert_equality_delete` can reuse the notifier returned by `try_start_eq_del_load` instead of minting and inserting a new one.



##########
crates/iceberg/src/arrow/delete_filter.rs:
##########
@@ -163,17 +163,22 @@ impl DeleteFilter {
         &self,
         file_path: &str,
     ) -> Option<Predicate> {
-        let notifier = {
+        // Create the `Notified` while holding the read lock. The read lock ensures that
+        // when we go inside it, either the state is already at Loaded or it is still at
+        // Loading AND `notify_waiters()` has not been called yet. Any `Notified` created
+        // before the invocation of `notify_waiters()` will be notified by it even if
+        // `await` has not been called on it yet.

Review Comment:
   The new comment explains the outcome (a `Notified` created before `notify_waiters()` still fires) but not the reason. The reason is specific: `notified_owned()` records tokio's internal `notify_waiters_calls` counter at construction, and on its first poll it completes if that counter has advanced. Holding the read lock is what guarantees the counter is read before the loader can advance it.
   
   Stating that keeps the code safe against a later refactor that moves `notified_owned()` out from under the lock, which would look harmless but would reintroduce the hang. The same point was made on #2859: https://github.com/apache/iceberg-rust/pull/2859#discussion_r3629462974
   
   Suggested wording:
   
   ```
   // Build the `Notified` while holding the read lock. `notified_owned()` records tokio's
   // `notify_waiters_calls` counter at construction and completes on first poll if that
   // counter has since advanced. Reading the counter under the lock guarantees it is taken
   // before `insert_equality_delete` can advance it via `notify_waiters()`, so the
   // notification is never missed even though we `.await` after releasing the lock.
   ```



##########
crates/iceberg/src/arrow/delete_filter.rs:
##########


Review Comment:
   Not a blocker, just raising it while we are here. On the error path, `parse_equality_deletes_record_batch_stream(...).await?` returns before `sender.send(predicate)`, dropping the sender. The spawned task's `eq_del.await.unwrap()` then panics on `RecvError`, so the `Loaded` insert and `notify_waiters()` never run and the entry stays `Loading`.
   
   Within one `load_deletes` call this looks contained: the error propagates via `item?` at `caching_delete_file_loader.rs:214`, the call returns `Err`, and siblings are cancelled rather than parked. It looks like a real hang only if two `load_deletes` calls share one `DeleteFilter` (concurrent scans on the same loader): the first marks the file `Loading`, fails, never signals, and a second reader parks forever.
   
   Two questions: is a shared `DeleteFilter` across concurrent loads a supported case? If so, would recording a failure state (or removing the entry so a reader retries) be preferable to leaving it `Loading` and panicking? Same shape as the positional path on #2859: https://github.com/apache/iceberg-rust/pull/2859#discussion_r3629462968.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: avoid lost-wakeup hang in get_equality_delete_predicate_for_delete_file_path [iceberg-rust]

Posted by "dhruvarya-db (via GitHub)" <gi...@apache.org>.
dhruvarya-db commented on PR #2873:
URL: https://github.com/apache/iceberg-rust/pull/2873#issuecomment-5805926636

   This PR is being revived in #3268.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org