You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "swanandx (via GitHub)" <gi...@apache.org> on 2026/09/24 21:52:43 UTC
[I] Scan stalls on an HTTP/2 object store when many files are read at once [iceberg-rust]
swanandx opened a new issue, #3276:
URL: https://github.com/apache/iceberg-rust/issues/3276
### Apache Iceberg Rust version
main, 0.10.1
### Describe the bug
A scan stops making progress and returns no error. We saw it stall against
GCS.
In `crates/iceberg/src/arrow/reader/pipeline.rs`:
```rust
tasks
.map_ok(move |task| task_reader.clone().process(task))
// .map_err(..) elided
.try_buffer_unordered(concurrency_limit_data_files)
.try_flatten_unordered(concurrency_limit_data_files)
```
There are two stages. `try_buffer_unordered` opens files, which means reading
each file's Parquet footer. `try_flatten_unordered` then reads the rows out.
The second stage handles at most `concurrency_limit_data_files` files at once.
When it is full it stops taking new ones, so the opens behind it still finish:
the footer arrives, and nobody ever reads it.
That is what breaks the scan. All these requests share one HTTP/2 connection,
and on such a connection the server may only send so far ahead of what the
client has read. HTTP/2 calls that allowance the connection window. A footer
nobody reads counts against the window and never gives it back. Once the unread
footers use up the window, the server cannot send anything more, including for
the files the scan is still reading. Everything stops, with no error.
Filling the window takes a lot of files open at once, which is why it shows
on big machines: `concurrency_limit_data_files` defaults to
`available_parallelism()`.
#806 introduced this, replacing a `spawn` and a bounded channel with these
combinators. In the old shape every file opened was being read.
### To Reproduce
It reproduces with no network, deterministically. Model one HTTP/2 connection
as a `Storage`: serve frames to the open streams round-robin, and size the
window to hold two footers. Scan twelve files, each larger than the window.
With `concurrency_limit_data_files` from 3 to 10 the scan stalls. It passes at
2, where the window still covers the unread footers, and at 11 and above, where
fewer than two files are left over to go unread. I have this as a test and will
attach it to a fix.
Two conditions have to hold, and both are defaults on a large machine reading a
table with more data files than cores:
- more data files than `concurrency_limit_data_files`, so the second stage fills
- enough unread footers to use up the window
We saw it against GCS on a 64 core machine, consistently. It did not reproduce
on 4 cores. We did not try the sizes in between, so I cannot give a threshold
in cores.
### Expected behavior
The scan finishes.
### Willingness to contribute
I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Scan stalls on an HTTP/2 object store when many files are read at once [iceberg-rust]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka closed issue #3276: Scan stalls on an HTTP/2 object store when many files are read at once
URL: https://github.com/apache/iceberg-rust/issues/3276
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org