You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "DerGut (via GitHub)" <gi...@apache.org> on 2026/07/28 22:17:58 UTC

[PR] Add RestSessionCatalog [iceberg-rust]

DerGut opened a new pull request, #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920

   ## Which issue does this PR close?
   
   - This is the second of ~3 PRs toward #2774.
   - Depends on #2836.
   
   ## What changes are included in this PR?
   
   This PR adds `RestSessionCatalog`, implementing the `SessionCatalog` API introduced in #2836.
   
   `RestSessionCatalog` becomes the core REST implementation. The existing `RestCatalog` delegates to it with an internal `SessionContext`, preserving session-unaware behavior. The REST test suite is refactored to exercise `RestSessionCatalog` directly.
   
   ## Are these changes tested?
   
   - `cargo test -p iceberg-catalog-rest --lib`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "DerGut (via GitHub)" <gi...@apache.org>.
DerGut commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3782179773


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -890,13 +1012,17 @@ impl Catalog for RestCatalog {
         }
     }
 
-    async fn namespace_exists(&self, ns: &NamespaceIdent) -> Result<bool> {
+    async fn namespace_exists(
+        &self,
+        context: &SessionContext,
+        ns: &NamespaceIdent,
+    ) -> Result<bool> {
         // Prefer a cheap HEAD when the server advertises it; otherwise fall back
         // to loading the namespace (GET) and treating a missing namespace as
         // `false`, so this still works against servers that don't advertise the
         // HEAD route.
         if !self.supports_endpoint(&V1_NAMESPACE_EXISTS).await? {

Review Comment:
   Updated in https://github.com/apache/iceberg-rust/pull/2920/commits/908c82058a8459ba5efcc9ebc23da8c205f9eb34



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "DerGut (via GitHub)" <gi...@apache.org>.
DerGut commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3782181004


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -419,6 +419,51 @@ struct RestClient {
 }
 
 impl RestClient {
+    async fn init(
+        user_config: &RestCatalogConfig,
+        auth_manager: Arc<dyn AuthManager>,
+    ) -> Result<Self> {
+        let http_client = HttpClient::new(user_config)?;
+        // The init session lives only for the config handshake, so a
+        // manager whose session guards a one-shot resource can release
+        // it before deriving the catalog session.
+        let catalog_config = {
+            let init_session = auth_manager
+                .init_session(
+                    &http_client.without_auth_session(),
+                    &RestSessionCatalog::auth_props(user_config),
+                )
+                .await?;
+            RestSessionCatalog::load_config(

Review Comment:
   Addressed with https://github.com/apache/iceberg-rust/pull/2920/commits/0ea8a83b9a676238ce1a75f91353ab7956d83158



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "DerGut (via GitHub)" <gi...@apache.org>.
DerGut commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3779321067


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -419,6 +419,51 @@ struct RestClient {
 }
 
 impl RestClient {
+    async fn init(
+        user_config: &RestCatalogConfig,
+        auth_manager: Arc<dyn AuthManager>,
+    ) -> Result<Self> {
+        let http_client = HttpClient::new(user_config)?;
+        // The init session lives only for the config handshake, so a
+        // manager whose session guards a one-shot resource can release
+        // it before deriving the catalog session.
+        let catalog_config = {
+            let init_session = auth_manager
+                .init_session(
+                    &http_client.without_auth_session(),
+                    &RestSessionCatalog::auth_props(user_config),
+                )
+                .await?;
+            RestSessionCatalog::load_config(

Review Comment:
   I'm leaning towards moving the two functions `RestSessionCatalog::auth_props` and `RestSessionCatalog::load_config` over to the `RestClient` too.
   
   They're only used in the `RestClient::init` and aren't part of the catalog API (not part of the `Catalog` trait).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "DerGut (via GitHub)" <gi...@apache.org>.
DerGut commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3782117509


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -4120,4 +4363,115 @@ mod tests {
             assert_eq!(err.message(), "Catalog uri is required");
         }
     }
+
+    #[tokio::test]
+    async fn test_create_session_catalog() {
+        let builder = RestSessionCatalogBuilder::default();
+
+        let result = builder
+            .load(
+                "test",
+                HashMap::from([
+                    (
+                        REST_CATALOG_PROP_URI.to_string(),
+                        "http://localhost:8080".to_string(),
+                    ),
+                    ("a".to_string(), "b".to_string()),
+                ]),
+            )
+            .await;
+
+        assert!(result.is_ok());
+
+        let catalog = result.unwrap();
+
+        let catalog_config = catalog.user_config;
+        assert_eq!(catalog_config.name.as_deref(), Some("test"));
+        assert_eq!(catalog_config.uri, "http://localhost:8080");
+        assert_eq!(catalog_config.warehouse, None);
+        // The default builder sets no client (only `with_client` does).
+        assert!(catalog_config.client.is_none());
+
+        // `uri` is consumed into its own field; other props are retained.
+        assert_eq!(catalog_config.props.get("a"), Some(&"b".to_string()));
+        assert!(!catalog_config.props.contains_key(REST_CATALOG_PROP_URI));
+    }
+
+    #[tokio::test]
+    async fn test_create_rest_catalog_with_session() {
+        let context = SessionContext::builder()
+            .session_id("test-id".to_string())
+            .build();
+
+        let result = RestCatalogBuilder::default()
+            .with_session_context(context)
+            .load(
+                "test",
+                HashMap::from([(
+                    REST_CATALOG_PROP_URI.to_string(),
+                    "http://localhost:8080".to_string(),
+                )]),
+            )
+            .await;
+
+        assert!(result.is_ok());
+
+        // The context passed to `with_session_context` is the one the catalog is bound to.
+        let catalog = result.unwrap();
+        assert_eq!(catalog.session_context.session_id(), "test-id");
+    }
+
+    #[tokio::test]
+    async fn test_create_rest_catalog_default_session() {
+        let result = RestCatalogBuilder::default()
+            .load(
+                "test",
+                HashMap::from([(
+                    REST_CATALOG_PROP_URI.to_string(),
+                    "http://localhost:8080".to_string(),
+                )]),
+            )
+            .await;
+
+        assert!(result.is_ok());
+
+        // Without `with_session`, the catalog falls back to `SessionContext::empty()`,

Review Comment:
   Updated in https://github.com/apache/iceberg-rust/pull/2920/commits/908c82058a8459ba5efcc9ebc23da8c205f9eb34



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY merged PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3780509428


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -4120,4 +4363,115 @@ mod tests {
             assert_eq!(err.message(), "Catalog uri is required");
         }
     }
+
+    #[tokio::test]
+    async fn test_create_session_catalog() {
+        let builder = RestSessionCatalogBuilder::default();
+
+        let result = builder
+            .load(
+                "test",
+                HashMap::from([
+                    (
+                        REST_CATALOG_PROP_URI.to_string(),
+                        "http://localhost:8080".to_string(),
+                    ),
+                    ("a".to_string(), "b".to_string()),
+                ]),
+            )
+            .await;
+
+        assert!(result.is_ok());
+
+        let catalog = result.unwrap();
+
+        let catalog_config = catalog.user_config;
+        assert_eq!(catalog_config.name.as_deref(), Some("test"));
+        assert_eq!(catalog_config.uri, "http://localhost:8080");
+        assert_eq!(catalog_config.warehouse, None);
+        // The default builder sets no client (only `with_client` does).
+        assert!(catalog_config.client.is_none());
+
+        // `uri` is consumed into its own field; other props are retained.
+        assert_eq!(catalog_config.props.get("a"), Some(&"b".to_string()));
+        assert!(!catalog_config.props.contains_key(REST_CATALOG_PROP_URI));
+    }
+
+    #[tokio::test]
+    async fn test_create_rest_catalog_with_session() {
+        let context = SessionContext::builder()
+            .session_id("test-id".to_string())
+            .build();
+
+        let result = RestCatalogBuilder::default()
+            .with_session_context(context)
+            .load(
+                "test",
+                HashMap::from([(
+                    REST_CATALOG_PROP_URI.to_string(),
+                    "http://localhost:8080".to_string(),
+                )]),
+            )
+            .await;
+
+        assert!(result.is_ok());
+
+        // The context passed to `with_session_context` is the one the catalog is bound to.
+        let catalog = result.unwrap();
+        assert_eq!(catalog.session_context.session_id(), "test-id");
+    }
+
+    #[tokio::test]
+    async fn test_create_rest_catalog_default_session() {
+        let result = RestCatalogBuilder::default()
+            .load(
+                "test",
+                HashMap::from([(
+                    REST_CATALOG_PROP_URI.to_string(),
+                    "http://localhost:8080".to_string(),
+                )]),
+            )
+            .await;
+
+        assert!(result.is_ok());
+
+        // Without `with_session`, the catalog falls back to `SessionContext::empty()`,

Review Comment:
   ```suggestion
           // Without `with_session_context`, the catalog falls back to `SessionContext::empty()`,
   ```



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -1083,7 +1219,11 @@ impl Catalog for RestCatalog {
     /// If there are any config properties that are present in both the response from the REST
     /// server and the config provided when creating this `RestCatalog` instance, then the value

Review Comment:
   The doc needs update here



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -419,6 +419,51 @@ struct RestClient {
 }
 
 impl RestClient {
+    async fn init(
+        user_config: &RestCatalogConfig,
+        auth_manager: Arc<dyn AuthManager>,
+    ) -> Result<Self> {
+        let http_client = HttpClient::new(user_config)?;
+        // The init session lives only for the config handshake, so a
+        // manager whose session guards a one-shot resource can release
+        // it before deriving the catalog session.
+        let catalog_config = {
+            let init_session = auth_manager
+                .init_session(
+                    &http_client.without_auth_session(),
+                    &RestSessionCatalog::auth_props(user_config),
+                )
+                .await?;
+            RestSessionCatalog::load_config(

Review Comment:
   makes sense to me, let's do it!



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -998,6 +1133,7 @@ impl Catalog for RestCatalog {
     /// the value provided locally to the `RestCatalog` will take precedence.

Review Comment:
   The doc here needs update



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -890,13 +1012,17 @@ impl Catalog for RestCatalog {
         }
     }
 
-    async fn namespace_exists(&self, ns: &NamespaceIdent) -> Result<bool> {
+    async fn namespace_exists(
+        &self,
+        context: &SessionContext,
+        ns: &NamespaceIdent,
+    ) -> Result<bool> {
         // Prefer a cheap HEAD when the server advertises it; otherwise fall back
         // to loading the namespace (GET) and treating a missing namespace as
         // `false`, so this still works against servers that don't advertise the
         // HEAD route.
         if !self.supports_endpoint(&V1_NAMESPACE_EXISTS).await? {

Review Comment:
   There are some documentation need to be updated since we are moving functions into RestSessionCatalog: https://github.com/apache/iceberg-rust/blob/main/crates/catalog/rest/src/endpoint.rs#L27



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "DerGut (via GitHub)" <gi...@apache.org>.
DerGut commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3786376755


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -1349,6 +1501,176 @@ impl Catalog for RestCatalog {
     }
 }
 
+/// Builder for an unbound [`RestSessionCatalog`].
+///
+/// Unlike [`RestCatalogBuilder`], the resulting catalog accepts a
+/// [`SessionContext`] with each [`SessionCatalog`] operation.
+#[derive(Debug)]
+pub struct RestSessionCatalogBuilder {

Review Comment:
   Good point! I have a local draft ready that I can clean up later. 
   
   The main open question IMO is how to support providers that were passed a `SessionCatalog` and ones that were just passed a `Catalog`.
   
   My current version uses a
   
   ```rust
   enum CatalogBackend {
       Bound(Arc<dyn Catalog>),
       Session(Arc<dyn SessionCatalog>, Arc<dyn SessionContextResolver>),
   }
   ```
   
   to share the Provider implementation and branch via the enum.
   
   `SessionContextResolver` will be something like
   
   ```rust
   pub trait SessionContextResolver {
     fn resolve_session_context(&self, state: DFSession) -> DFResult<SessionContext>;
   }
   ```
   
   because Datafusion doesn't have a canonical way to represent query context like Trino does (beyond its `Session.session_id()`). Users instead pass metadata via extensions which they will need to extract in custom implementations and convert to `SessionContext` themselves.
   
   More on this hopefully this weekend!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3786269941


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -1349,6 +1501,176 @@ impl Catalog for RestCatalog {
     }
 }
 
+/// Builder for an unbound [`RestSessionCatalog`].
+///
+/// Unlike [`RestCatalogBuilder`], the resulting catalog accepts a
+/// [`SessionContext`] with each [`SessionCatalog`] operation.
+#[derive(Debug)]
+pub struct RestSessionCatalogBuilder {

Review Comment:
   How do we plan to integrate SessionCatalog with DataFusion? I saw you mentioned that we can use SessionCatalog in IcebergTableProvider in the issue, but I'm not sure with that change, how users will use other non-rest catalogs.
   
   Maybe it's better to put out a draft PR for that integration, and we can discuss there?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "DerGut (via GitHub)" <gi...@apache.org>.
DerGut commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3722313850


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -60,105 +60,56 @@ const CARGO_PKG_VERSION: &str = env!("CARGO_PKG_VERSION");
 const PATH_V1: &str = "v1";
 
 /// Builder for [`RestCatalog`].
-#[derive(Debug)]
+#[derive(Debug, Default)]
 pub struct RestCatalogBuilder {
-    config: RestCatalogConfig,
-    storage_factory: Option<Arc<dyn StorageFactory>>,
-    kms_client_factory: Option<Arc<dyn KmsClientFactory>>,
-    runtime: Option<Runtime>,
-}
-
-impl Default for RestCatalogBuilder {
-    fn default() -> Self {
-        Self {
-            config: RestCatalogConfig {
-                name: None,
-                uri: "".to_string(),
-                warehouse: None,
-                props: HashMap::new(),
-                client: None,
-            },
-            storage_factory: None,
-            kms_client_factory: None,
-            runtime: None,
-        }
-    }
+    session: Option<SessionContext>,
+    inner: RestSessionCatalogBuilder,
 }
 
 impl CatalogBuilder for RestCatalogBuilder {
     type C = RestCatalog;
 
     fn with_storage_factory(mut self, storage_factory: Arc<dyn StorageFactory>) -> Self {
-        self.storage_factory = Some(storage_factory);
+        self.inner = self.inner.with_storage_factory(storage_factory);
         self
     }
 
     fn with_kms_client_factory(mut self, kms_client_factory: Arc<dyn KmsClientFactory>) -> Self {
-        self.kms_client_factory = Some(kms_client_factory);
+        self.inner = self.inner.with_kms_client_factory(kms_client_factory);
         self
     }
 
     fn with_runtime(mut self, runtime: Runtime) -> Self {
-        self.runtime = Some(runtime);
+        self.inner = self.inner.with_runtime(runtime);
         self
     }
 
     fn load(
-        mut self,
+        self,
         name: impl Into<String>,
         props: HashMap<String, String>,
     ) -> impl Future<Output = Result<Self::C>> + Send {
-        self.config.name = Some(name.into());
-
-        if props.contains_key(REST_CATALOG_PROP_URI) {
-            self.config.uri = props
-                .get(REST_CATALOG_PROP_URI)
-                .cloned()
-                .unwrap_or_default();
-        }
-
-        if props.contains_key(REST_CATALOG_PROP_WAREHOUSE) {
-            self.config.warehouse = props.get(REST_CATALOG_PROP_WAREHOUSE).cloned()
-        }
-
-        // Collect other remaining properties
-        self.config.props = props
-            .into_iter()
-            .filter(|(k, _)| k != REST_CATALOG_PROP_URI && k != REST_CATALOG_PROP_WAREHOUSE)
-            .collect();
-
+        let name = name.into();
         async move {
-            if self.config.name.is_none() {
-                Err(Error::new(
-                    ErrorKind::DataInvalid,
-                    "Catalog name is required",
-                ))
-            } else if self.config.uri.is_empty() {
-                Err(Error::new(
-                    ErrorKind::DataInvalid,
-                    "Catalog uri is required",
-                ))
-            } else {
-                let runtime = self.runtime.unwrap_or_else(Runtime::current);
-                let kms_client = match self.kms_client_factory {
-                    Some(factory) => Some(factory.create_kms_client(&self.config.props).await?),
-                    None => None,
-                };
-                Ok(RestCatalog::new(
-                    self.config,
-                    self.storage_factory,
-                    runtime,
-                    kms_client,
-                ))
-            }
+            let session = self.session.unwrap_or_else(SessionContext::empty);
+            let session_catalog = Arc::new(self.inner.load(name, props).await?);
+
+            Ok(RestCatalog::from_session_catalog(session, session_catalog))
         }
     }
 }
 
 impl RestCatalogBuilder {
     /// Configures the catalog with a custom HTTP client.
     pub fn with_client(mut self, client: Client) -> Self {
-        self.config.client = Some(client);
+        self.inner = self.inner.with_client(client);
+        self
+    }
+
+    /// Configures the session that will be used with this catalog.
+    /// Overwrites the default empty session from SessionContext::empty().
+    pub fn with_session(mut self, session: SessionContext) -> Self {

Review Comment:
   As mentioned in the "Welcome Feedback ❓" section, I could use some opinions on the `session` vs. `context`/ `ctx` variable naming. I feel like what we introduce here will be reused throughout.
   
   ```suggestion
       pub fn with_session_context(mut self, ctx: SessionContext) -> Self {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "DerGut (via GitHub)" <gi...@apache.org>.
DerGut commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3786376755


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -1349,6 +1501,176 @@ impl Catalog for RestCatalog {
     }
 }
 
+/// Builder for an unbound [`RestSessionCatalog`].
+///
+/// Unlike [`RestCatalogBuilder`], the resulting catalog accepts a
+/// [`SessionContext`] with each [`SessionCatalog`] operation.
+#[derive(Debug)]
+pub struct RestSessionCatalogBuilder {

Review Comment:
   Good point! I have a local draft ready that I can clean up later. 
   
   The main open question IMO is how to support providers that were passed a `SessionCatalog` and ones that were just passed a `Catalog`.
   
   My current version uses a
   
   ```rust
   enum CatalogBackend {
       Bound(Arc<dyn Catalog>),
       Session(Arc<dyn SessionCatalog>, Arc<dyn SessionContextResolver>),
   }
   ```
   
   Where `SessionContextResolver` will be something like
   
   ```rust
   pub trait SessionContextResolver {
     fn resolve_session_context(&self, state: DFSession) -> DFResult<SessionContext>;
   }
   ```
   
   because Datafusion doesn't have a canonical way to represent query context like Trino does (beyond its `Session.session_id()`). Users instead pass metadata via extensions which they will need to extract in custom implementations and convert to `SessionContext` themselves.
   
   More on this hopefully this weekend!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3777708718


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -60,105 +60,56 @@ const CARGO_PKG_VERSION: &str = env!("CARGO_PKG_VERSION");
 const PATH_V1: &str = "v1";
 
 /// Builder for [`RestCatalog`].
-#[derive(Debug)]
+#[derive(Debug, Default)]
 pub struct RestCatalogBuilder {
-    config: RestCatalogConfig,
-    storage_factory: Option<Arc<dyn StorageFactory>>,
-    kms_client_factory: Option<Arc<dyn KmsClientFactory>>,
-    runtime: Option<Runtime>,
-}
-
-impl Default for RestCatalogBuilder {
-    fn default() -> Self {
-        Self {
-            config: RestCatalogConfig {
-                name: None,
-                uri: "".to_string(),
-                warehouse: None,
-                props: HashMap::new(),
-                client: None,
-            },
-            storage_factory: None,
-            kms_client_factory: None,
-            runtime: None,
-        }
-    }
+    session: Option<SessionContext>,
+    inner: RestSessionCatalogBuilder,
 }
 
 impl CatalogBuilder for RestCatalogBuilder {
     type C = RestCatalog;
 
     fn with_storage_factory(mut self, storage_factory: Arc<dyn StorageFactory>) -> Self {
-        self.storage_factory = Some(storage_factory);
+        self.inner = self.inner.with_storage_factory(storage_factory);
         self
     }
 
     fn with_kms_client_factory(mut self, kms_client_factory: Arc<dyn KmsClientFactory>) -> Self {
-        self.kms_client_factory = Some(kms_client_factory);
+        self.inner = self.inner.with_kms_client_factory(kms_client_factory);
         self
     }
 
     fn with_runtime(mut self, runtime: Runtime) -> Self {
-        self.runtime = Some(runtime);
+        self.inner = self.inner.with_runtime(runtime);
         self
     }
 
     fn load(
-        mut self,
+        self,
         name: impl Into<String>,
         props: HashMap<String, String>,
     ) -> impl Future<Output = Result<Self::C>> + Send {
-        self.config.name = Some(name.into());
-
-        if props.contains_key(REST_CATALOG_PROP_URI) {
-            self.config.uri = props
-                .get(REST_CATALOG_PROP_URI)
-                .cloned()
-                .unwrap_or_default();
-        }
-
-        if props.contains_key(REST_CATALOG_PROP_WAREHOUSE) {
-            self.config.warehouse = props.get(REST_CATALOG_PROP_WAREHOUSE).cloned()
-        }
-
-        // Collect other remaining properties
-        self.config.props = props
-            .into_iter()
-            .filter(|(k, _)| k != REST_CATALOG_PROP_URI && k != REST_CATALOG_PROP_WAREHOUSE)
-            .collect();
-
+        let name = name.into();
         async move {
-            if self.config.name.is_none() {
-                Err(Error::new(
-                    ErrorKind::DataInvalid,
-                    "Catalog name is required",
-                ))
-            } else if self.config.uri.is_empty() {
-                Err(Error::new(
-                    ErrorKind::DataInvalid,
-                    "Catalog uri is required",
-                ))
-            } else {
-                let runtime = self.runtime.unwrap_or_else(Runtime::current);
-                let kms_client = match self.kms_client_factory {
-                    Some(factory) => Some(factory.create_kms_client(&self.config.props).await?),
-                    None => None,
-                };
-                Ok(RestCatalog::new(
-                    self.config,
-                    self.storage_factory,
-                    runtime,
-                    kms_client,
-                ))
-            }
+            let session = self.session.unwrap_or_else(SessionContext::empty);
+            let session_catalog = Arc::new(self.inner.load(name, props).await?);
+
+            Ok(RestCatalog::from_session_catalog(session, session_catalog))
         }
     }
 }
 
 impl RestCatalogBuilder {
     /// Configures the catalog with a custom HTTP client.
     pub fn with_client(mut self, client: Client) -> Self {
-        self.config.client = Some(client);
+        self.inner = self.inner.with_client(client);
+        self
+    }
+
+    /// Configures the session that will be used with this catalog.
+    /// Overwrites the default empty session from SessionContext::empty().
+    pub fn with_session(mut self, session: SessionContext) -> Self {

Review Comment:
   This is a good point and I agree that naming is important, I'm thinking of:
   
   session_context or context: SessionContext
   auth_session: AuthSession (some existing fields are named `session` which I think can be more explicit)
   RestContext -> RestClient sounds like a good direction to address the naming conflict
   
   I think a small PR to address the naming before the feature change would be very helpful



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "DerGut (via GitHub)" <gi...@apache.org>.
DerGut commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3778088418


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -60,105 +60,56 @@ const CARGO_PKG_VERSION: &str = env!("CARGO_PKG_VERSION");
 const PATH_V1: &str = "v1";
 
 /// Builder for [`RestCatalog`].
-#[derive(Debug)]
+#[derive(Debug, Default)]
 pub struct RestCatalogBuilder {
-    config: RestCatalogConfig,
-    storage_factory: Option<Arc<dyn StorageFactory>>,
-    kms_client_factory: Option<Arc<dyn KmsClientFactory>>,
-    runtime: Option<Runtime>,
-}
-
-impl Default for RestCatalogBuilder {
-    fn default() -> Self {
-        Self {
-            config: RestCatalogConfig {
-                name: None,
-                uri: "".to_string(),
-                warehouse: None,
-                props: HashMap::new(),
-                client: None,
-            },
-            storage_factory: None,
-            kms_client_factory: None,
-            runtime: None,
-        }
-    }
+    session: Option<SessionContext>,
+    inner: RestSessionCatalogBuilder,
 }
 
 impl CatalogBuilder for RestCatalogBuilder {
     type C = RestCatalog;
 
     fn with_storage_factory(mut self, storage_factory: Arc<dyn StorageFactory>) -> Self {
-        self.storage_factory = Some(storage_factory);
+        self.inner = self.inner.with_storage_factory(storage_factory);
         self
     }
 
     fn with_kms_client_factory(mut self, kms_client_factory: Arc<dyn KmsClientFactory>) -> Self {
-        self.kms_client_factory = Some(kms_client_factory);
+        self.inner = self.inner.with_kms_client_factory(kms_client_factory);
         self
     }
 
     fn with_runtime(mut self, runtime: Runtime) -> Self {
-        self.runtime = Some(runtime);
+        self.inner = self.inner.with_runtime(runtime);
         self
     }
 
     fn load(
-        mut self,
+        self,
         name: impl Into<String>,
         props: HashMap<String, String>,
     ) -> impl Future<Output = Result<Self::C>> + Send {
-        self.config.name = Some(name.into());
-
-        if props.contains_key(REST_CATALOG_PROP_URI) {
-            self.config.uri = props
-                .get(REST_CATALOG_PROP_URI)
-                .cloned()
-                .unwrap_or_default();
-        }
-
-        if props.contains_key(REST_CATALOG_PROP_WAREHOUSE) {
-            self.config.warehouse = props.get(REST_CATALOG_PROP_WAREHOUSE).cloned()
-        }
-
-        // Collect other remaining properties
-        self.config.props = props
-            .into_iter()
-            .filter(|(k, _)| k != REST_CATALOG_PROP_URI && k != REST_CATALOG_PROP_WAREHOUSE)
-            .collect();
-
+        let name = name.into();
         async move {
-            if self.config.name.is_none() {
-                Err(Error::new(
-                    ErrorKind::DataInvalid,
-                    "Catalog name is required",
-                ))
-            } else if self.config.uri.is_empty() {
-                Err(Error::new(
-                    ErrorKind::DataInvalid,
-                    "Catalog uri is required",
-                ))
-            } else {
-                let runtime = self.runtime.unwrap_or_else(Runtime::current);
-                let kms_client = match self.kms_client_factory {
-                    Some(factory) => Some(factory.create_kms_client(&self.config.props).await?),
-                    None => None,
-                };
-                Ok(RestCatalog::new(
-                    self.config,
-                    self.storage_factory,
-                    runtime,
-                    kms_client,
-                ))
-            }
+            let session = self.session.unwrap_or_else(SessionContext::empty);
+            let session_catalog = Arc::new(self.inner.load(name, props).await?);
+
+            Ok(RestCatalog::from_session_catalog(session, session_catalog))
         }
     }
 }
 
 impl RestCatalogBuilder {
     /// Configures the catalog with a custom HTTP client.
     pub fn with_client(mut self, client: Client) -> Self {
-        self.config.client = Some(client);
+        self.inner = self.inner.with_client(client);
+        self
+    }
+
+    /// Configures the session that will be used with this catalog.
+    /// Overwrites the default empty session from SessionContext::empty().
+    pub fn with_session(mut self, session: SessionContext) -> Self {

Review Comment:
   Rename PR: https://github.com/apache/iceberg-rust/pull/2994



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "DerGut (via GitHub)" <gi...@apache.org>.
DerGut commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3782167133


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -1083,7 +1219,11 @@ impl Catalog for RestCatalog {
     /// If there are any config properties that are present in both the response from the REST
     /// server and the config provided when creating this `RestCatalog` instance, then the value

Review Comment:
   Updated in https://github.com/apache/iceberg-rust/pull/2920/commits/908c82058a8459ba5efcc9ebc23da8c205f9eb34



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -998,6 +1133,7 @@ impl Catalog for RestCatalog {
     /// the value provided locally to the `RestCatalog` will take precedence.

Review Comment:
   Updated in https://github.com/apache/iceberg-rust/pull/2920/commits/908c82058a8459ba5efcc9ebc23da8c205f9eb34



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "DerGut (via GitHub)" <gi...@apache.org>.
DerGut commented on PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#issuecomment-5291143557

   Thanks for the quick review!
   I've pushed a commit with documentation changes which should address all your comments. It also includes an example file that's now referenced from the website docs.
   
   > Also I'm assuming we will actually propagate the context in the followup PR?
   
   Exactly! I'll push a draft very soon 🚀 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add RestSessionCatalog [iceberg-rust]

Posted by "DerGut (via GitHub)" <gi...@apache.org>.
DerGut commented on code in PR #2920:
URL: https://github.com/apache/iceberg-rust/pull/2920#discussion_r3789981475


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -1349,6 +1501,176 @@ impl Catalog for RestCatalog {
     }
 }
 
+/// Builder for an unbound [`RestSessionCatalog`].
+///
+/// Unlike [`RestCatalogBuilder`], the resulting catalog accepts a
+/// [`SessionContext`] with each [`SessionCatalog`] operation.
+#[derive(Debug)]
+pub struct RestSessionCatalogBuilder {

Review Comment:
   This is the draft to add the Datafusion integration: https://github.com/apache/iceberg-rust/pull/3000



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org