You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "huan233usc (via GitHub)" <gi...@apache.org> on 2026/06/16 05:22:45 UTC
[PR] feat(catalog-rest): client support for server-side REST scan planning [iceberg-rust]
huan233usc opened a new pull request, #2656:
URL: https://github.com/apache/iceberg-rust/pull/2656
## Which issue does this PR close?
Part of #1690 — client support for REST server-side scan planning.
## What changes are included in this PR?
Adds a client implementation of the REST scan-planning protocol
(`planTableScan` / `fetchPlanningResult` / `fetchScanTasks`). When a catalog
advertises the planning endpoints, a table scan delegates planning to the
server and consumes the returned `FileScanTask`s instead of reading manifests
locally; otherwise it transparently falls back to native client-side planning.
## Design
The whole feature hangs off a single seam — `TableScan::plan_files()` — so
execution (`to_arrow`, the Arrow reader) is untouched and DataFusion needs no
changes.
```
Table::scan() ─► TableScan::plan_files()
│ injected ScanPlanner present?
├─ no ─► native manifest planning (unchanged)
└─ yes ─► ScanPlanner::plan_table_scan(ScanPlanRequest)
│
┌───────┴─ endpoint negotiation: gate on the endpoints advertised
│ by GET /v1/config (else FeatureUnsupported → native)
▼
POST .../plan ──► COMPLETED ─────────────────────────────┐
└─► SUBMITTED ─► poll GET .../plan/{id} │
(exp backoff 1s→60s) ──────┤ COMPLETED
plan-tasks? ─► POST .../tasks (tokens may recurse) ────────┤
▼
convert wire content-files ─► FileScanTask (public builders)
+ build a plan-scoped FileIO from vended `storage-credentials`
▼
ServerScanPlan { tasks, file_io } ─► to_arrow() reads tasks through file_io
(on Drop before completion: best-effort DELETE .../plan/{id})
```
### Components / flow
1. **Injection seam** — a narrow `ScanPlanner` capability trait
(`crates/iceberg/src/scan/planner.rs`). `Table`/`TableScanBuilder` carry an
optional `Arc<dyn ScanPlanner>`; `plan_files()` delegates to it and falls
back to native planning on `ErrorKind::FeatureUnsupported`. The core
`Catalog` trait is untouched.
2. **Endpoint negotiation** — `CatalogConfig` now parses the `endpoints` field
of `GET /v1/config`; the scan-plan calls are gated by an `Endpoint::check`.
3. **Wire DTOs** — request/response types for plan / fetch-planning-result /
fetch-scan-tasks, plus a lean content-file shape (only the fields a
`FileScanTask` needs).
4. **State machine** — submit → poll-with-backoff → fan-out `fetchScanTasks`
(plan-task tokens may produce more tasks), with a best-effort
`DELETE .../plan/{id}` if the scan is dropped mid-flight.
5. **Conversion** — wire content-files → `FileScanTask` via the public
builders (no `DataFile` internals); the scan's own bound filter is used as
the per-task row predicate, and pushed down as Iceberg expression JSON when
losslessly encodable.
6. **Credential vending** — `ScanPlanner::plan_table_scan` returns
`ServerScanPlan { tasks, file_io }`; the planner builds a plan-scoped
`FileIO` from the `storage-credentials` the server returns, and `to_arrow`
reads data files through it.
### Alternative injection design
The same feature with the planning capability placed on the core `Catalog`
trait (`Catalog::plan_table_scan`, `Table` holding `Arc<dyn Catalog>`) instead
of a narrow `ScanPlanner` trait is in
https://github.com/huan233usc/iceberg-rust/pull/2 for comparison. This PR uses
the narrow-trait design because it keeps the central `Catalog` trait minimal
and avoids giving every `Table` a back-reference to the full catalog.
## Are these changes tested?
Yes — unit tests for the wire DTOs, endpoint codec, and expression-JSON
serialization, conversion tests, and end-to-end `mockito` tests covering the
completed-inline, submitted-then-polled, and recursive plan-task fan-out paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(catalog-rest): client support for server-side REST scan planning [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2656: feat(catalog-rest): client support for server-side REST scan planning
URL: https://github.com/apache/iceberg-rust/pull/2656
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(catalog-rest): client support for server-side REST scan planning [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2656:
URL: https://github.com/apache/iceberg-rust/pull/2656#issuecomment-5064876306
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(catalog-rest): client support for server-side REST scan planning [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2656:
URL: https://github.com/apache/iceberg-rust/pull/2656#issuecomment-5137757851
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org