You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "bcgraham (via GitHub)" <gi...@apache.org> on 2026/08/07 18:36:00 UTC
[I] opendal FileIO hard-wires TimeoutLayer defaults (10s per-IO) with no way to configure them; large Parquet row groups fail deterministically on S3 [iceberg-rust]
bcgraham opened a new issue, #2977:
URL: https://github.com/apache/iceberg-rust/issues/2977
### Apache Iceberg Rust version
Reproduced on `iceberg` 0.10.1 / `iceberg-storage-opendal` 0.10.1. The code path is unchanged on `main` as of 2026-08-07 (`86d9d7d`).
### Describe the bug
`iceberg-storage-opendal` wraps every FileIO operator in `TimeoutLayer::new()`, and nothing in the crate's property or builder surface can override it:
https://github.com/apache/iceberg-rust/blob/86d9d7dc4297302496e6a121934de719c3ba9bea/crates/storage/opendal/src/lib.rs#L368
`TimeoutLayer::new()` uses opendal's defaults — **60 s overall, 10 s per IO operation** ([opendal 0.58 docs](https://docs.rs/opendal/0.58.0/opendal/layers/struct.TimeoutLayer.html)).
The 10 s `io_timeout` is the one that bites on the write path. opendal turns a single large `write` into a single S3 multipart part, and parquet-rs flushes a completed row group as one write — so in practice **the row group is the part**. Any row group whose encoded size exceeds what the writer's uplink can push in 10 s fails, and fails *deterministically*: the same request is retried by the `RetryLayer` above it and cannot ever fit inside the budget, so every attempt dies at the same place.
This is easy to hit on a table with a wide binary/string column, where row-group size is dominated by a handful of large values rather than by the row count the caller configured.
### To Reproduce
Environment: `iceberg` 0.10.1 + `iceberg-catalog-glue`, `parquet`/`arrow` 58, rustc 1.97.1, S3 over a WAN link (developer laptop, not in-region).
1. Create an Iceberg table with a `binary` payload column carrying large values — in our corpus ~9,200 rows / ~1.3 GB total, p50 ≈ 2.9 KB, p99 ≈ 1.4 MB, max ≈ 33 MB.
2. Write it through `ParquetWriterBuilder` with row-count row-group sizing, `WriterProperties::set_max_row_group_size(2048)`.
3. The write fails while flushing a row group:
```
Unexpected (persistent) at write, context: {
upload_id: <redacted>, part_number: 0, service: s3,
path: <table>/data/00000-....parquet,
size: 125382156, written: 428988086
} => write part timeout
1: External: External: Unexpected => Failure in doing io operation, ...
...
4: Unexpected (persistent) at write, context: { timeout: 10 } => io operation timeout reached
```
`size: 125382156` is the part being flushed; `timeout: 10` is the hard-wired `io_timeout`. **Three separate runs of the identical configuration failed at the identical byte offset.**
Two controls isolate the cause:
* **Same data, same row-group configuration, written with `arrow-rs` + `object_store` directly** (no Iceberg FileIO): completed in 44 s, no error. So this is not parquet, not arrow, and not S3.
* **Same data through Iceberg with smaller row groups** (512 rows, ≈4× smaller parts; and byte-based row-group targets of 4/16/64 MiB set in code): all succeed.
The threshold itself is bandwidth-dependent — an in-region task will tolerate a much larger row group than this laptop did — but the *ceiling exists at every bandwidth* and is not visible or adjustable from the Iceberg API.
### Expected behavior
Either of:
1. **Make the timeouts configurable.** A FileIO property in the existing `client.*` namespace (alongside `client.region`, `client.assume-role.arn`, …) — e.g. `client.io-timeout-ms` / `client.timeout-ms` — or a builder hook on the storage config, so a caller writing large parts can raise the per-IO budget.
2. **Scale the per-IO timeout with the size of the write being attempted**, so a large multipart part gets a proportional budget instead of a fixed 10 s.
The current default is a sensible one, and the comment at the call site is right that `TimeoutLayer` must sit inside `RetryLayer` so each attempt is independently bounded. The problem is only that it is a *hard* ceiling: it silently caps the maximum row-group size a table can be written with, as a function of the writer's uplink bandwidth rather than of anything the table or the data expresses — and it surfaces as a persistent, retry-proof write failure rather than as a configuration error.
### Willingness to contribute
I cannot contribute a fix for this bug at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] opendal FileIO hard-wires TimeoutLayer defaults (10s per-IO) with no way to configure them; large Parquet row groups fail deterministically on S3 [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #2977:
URL: https://github.com/apache/iceberg-rust/issues/2977#issuecomment-5780098871
In Comet we also see this problem and would like to see the way to configure the timeout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] opendal FileIO hard-wires TimeoutLayer defaults (10s per-IO) with no way to configure them; large Parquet row groups fail deterministically on S3 [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on issue #2977:
URL: https://github.com/apache/iceberg-rust/issues/2977#issuecomment-5335021839
Take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] opendal FileIO hard-wires TimeoutLayer defaults (10s per-IO) with no way to configure them; large Parquet row groups fail deterministically on S3 [iceberg-rust]
Posted by "YuangGao (via GitHub)" <gi...@apache.org>.
YuangGao commented on issue #2977:
URL: https://github.com/apache/iceberg-rust/issues/2977#issuecomment-5597375144
Opened https://github.com/apache/iceberg-rust/pull/3179 — instead of loosening the deadline it bounds the request, since opendal sends one request per caller buffer and `ParquetWriter` hands it a whole row group at a time.
Heads-up on option 2 though: opendal dropped `TimeoutLayer::with_speed` in apache/opendal#6793, so scaling the deadline with payload size isn't really available anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org