You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "DerGut (via GitHub)" <gi...@apache.org> on 2026/07/06 13:36:13 UTC
[I] Session/ Request/ Auth Context [iceberg-rust]
DerGut opened a new issue, #2774:
URL: https://github.com/apache/iceberg-rust/issues/2774
### Is your feature request related to a problem or challenge?
We need to pass request context from Datafusion to our Iceberg connector.
Today, the [Datafusion Session](https://github.com/apache/datafusion/blob/6a0e76ed692afaafd616e0d98d51bef63c1ff02d/datafusion/session/src/session.rs#L75) (which is already available for each query) terminates at the Iceberg catalog boundary. Scans for example, use a shared catalog and then [call the `catalog.load_table(&self.table_ident)`](https://github.com/apache/iceberg-rust/blob/22e80a63dbb4e58e034672b4ff665a6eae82fa02/crates/integrations/datafusion/src/table/mod.rs#L135) function on it, which doesn't support any way of context propagation.
Our downstream REST catalog requires this context to make authorization, rate limiting and shard routing decisions.
### Describe the solution you'd like
### SessionCatalog Trait
The Java implementation solves this with a separate [`SessionCatalog`](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/catalog/SessionCatalog.java#L34) interface which coexists next to the regular `Catalog` interface.
It replicates all interface methods, but accepts a [`SessionContext`](https://github.com/apache/iceberg/blob/2d3f5c24e0697b361fe95362d8ee13b915e98048/api/src/main/java/org/apache/iceberg/catalog/SessionCatalog.java#L36) for each. Compare the Java [Catalog's](https://github.com/apache/iceberg/blob/2d3f5c24e0697b361fe95362d8ee13b915e98048/api/src/main/java/org/apache/iceberg/catalog/Catalog.java#L326):
```java
Table loadTable(TableIdentifier ident);
```
to the [SessionCatalog's](https://github.com/apache/iceberg/blob/2d3f5c24e0697b361fe95362d8ee13b915e98048/api/src/main/java/org/apache/iceberg/catalog/SessionCatalog.java#L223):
```java
Table loadTable(SessionContext context, TableIdentifier ident);
```
A Rust implementation could re-use this mechanism and provide a separate `SessionCatalog` trait, along with a `SessionContext` struct (which is a simple struct without any behavior). The Datafusion integration's `IcebergTableProvider` could then store an [`Arc<dyn SessionCatalog>` instead](https://github.com/apache/iceberg-rust/blob/22e80a63dbb4e58e034672b4ff665a6eae82fa02/crates/integrations/datafusion/src/table/mod.rs#L71), translate the Datafusion `Session` to an Iceberg `SessionContext` for each query, and pass it along to the Catalog.
### REST Implementation
The RestCatalog then needs to materialize the query context into the HTTP request metadata. For our use case, we really only need to forward the same context that we received, but the full Java implementation allows for authentication handled by the client.
The Java implementation uses the [AuthManager](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/auth/AuthManager.java#L33) for this. It is a field of the RestSessionCatalog implementation, and is [called on each catalog method](https://github.com/apache/iceberg/blob/2d3f5c24e0697b361fe95362d8ee13b915e98048/core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java#L336) to produce a "contextual" AuthSession from a SessionContext.
The `AuthSession` is then used by the Rest/ HTTP client to authenticate any request with headers.
Note that this flow doesn't support propagation of arbitrary metadata, but only authentication information supported by the AuthManager/ AuthContext. This still suits our current needs.
---
On my willingness to contribute: I'm happy to handle the implementation, but I need some feedback on the overall design (whether or not we want to follow the Java approach).
### Willingness to contribute
I would be willing to contribute to this feature with guidance from the Iceberg Rust community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Session/ Request/ Auth Context Propagation [iceberg-rust]
Posted by "DerGut (via GitHub)" <gi...@apache.org>.
DerGut commented on issue #2774:
URL: https://github.com/apache/iceberg-rust/issues/2774#issuecomment-4996531825
I've sketched out a plan to implement this work in 3 phases:
1. Introduce a `SessionCatalog` following Java's SessionCatalog interface, have it accept a `SessionContext` for catalog methods. The `RestCatalog` can simply reuse it as an inner catalog and delegate all logic -> PoC is here: https://github.com/apache/iceberg-rust/pull/2836
2. Introduce `AuthManager` and `AuthSession`. Then allow the `RestCatalog`+`RestSessionCatalog` to hold a manager, and use it to translate `SessionContext`s to `AuthSession`s. Introduce caching for auth sessions at catalog and table level. Add 1-2 simple reference implementations (e.g. `BasicAuthManager`) -> to be PoC'd but building on prior work from https://github.com/apache/iceberg-rust/pull/1512
3. Wire the `SessionCatalog` into the Datafusion integration crate. Translate `datafusion::catalog::Session`s to `iceberg::SessionContext`s and propagate them to catalog functions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org