You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "connortsui20 (via GitHub)" <gi...@apache.org> on 2025/02/12 17:19:38 UTC

[PR] refactor: REST `Catalog` implementation [iceberg-rust]

connortsui20 opened a new pull request, #965:
URL: https://github.com/apache/iceberg-rust/pull/965

   (STILL A DRAFT)
   
   Followup of #962
   
   #962 Introduced a bug where it is not some of the methods allow for both `StatusCode::OK` and `StatusCode::NO_CONTENT` as success cases, when in reality it should be one or the other (this was me, sorry about that).
   
   This PR attempts to unify the 3 different types of response helpers that essentially all do the exact same thing slightly differently. The main addition here is a function `query_catalog`:
   
   ```rust
       // Queries the Iceberg REST catalog with the given `Request` and a provided handler.
       pub async fn query_catalog<R, H, Fut>(&self, mut request: Request, handler: H) -> Result<R>
       where
           R: DeserializeOwned,
           H: FnOnce(Response) -> Fut,
           Fut: Future<Output = Result<R>>,
       {
           self.authenticate(&mut request).await?;
   
           let response = self.client.execute(request).await?;
   
           handler(response).await
       }
   ```
   
   By allowing each `Catalog` method to specify how they want to handle the responses, it gets much finer control on the success/error cases as well as the error messages. Previously, there were 3 functions that all did similar things:
   
   ```rust
       pub async fn query<R: DeserializeOwned, E: DeserializeOwned + Into<Error>>(
           &self,
           mut request: Request,
       ) -> Result<R> {
   
       pub async fn execute<E: DeserializeOwned + Into<Error>>(
           &self,
           mut request: Request,
       ) -> Result<()> {
   
       pub async fn do_execute<R, E: DeserializeOwned + Into<Error>>(
           &self,
           mut request: Request,
           handler: impl FnOnce(&Response) -> Option<R>,
       ) -> Result<R> {
   ```
   
   I'm also somewhat using this as a chance to refactor some other parts of this crate, mainly documentation and examples.
   
   (DRAFT)
   
   @Xuanwo It would be great if I could get feedback on some of these proposed changes before I keep going!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "connortsui20 (via GitHub)" <gi...@apache.org>.
connortsui20 commented on PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#issuecomment-2692409369

   @Xuanwo ready for review! I think that this version is a lot more boilerplate but that should be fine as it is more or less linearly readable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#issuecomment-2693941133

   Inviting @liurenjie1024 for another look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "connortsui20 (via GitHub)" <gi...@apache.org>.
connortsui20 commented on code in PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1977962499


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -312,90 +317,107 @@ impl RestCatalog {
     }
 }
 
+/// All requests and expected responses are derived from the REST catalog API spec:
+/// https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml
 #[async_trait]
 impl Catalog for RestCatalog {
-    /// List namespaces from table.
     async fn list_namespaces(
         &self,
         parent: Option<&NamespaceIdent>,
     ) -> Result<Vec<NamespaceIdent>> {
-        let mut request = self.context().await?.client.request(
-            Method::GET,
-            self.context().await?.config.namespaces_endpoint(),
-        );
-        if let Some(ns) = parent {
-            request = request.query(&[("parent", ns.to_url_string())]);
-        }
+        let context = self.context().await?;
 
-        let resp = self
-            .context()
-            .await?
+        let mut request_builder = context
             .client
-            .query::<ListNamespaceResponse, ErrorResponse>(request.build()?)
-            .await?;
+            .request(Method::GET, context.config.namespaces_endpoint());
+        // Filter on `parent={namespace}` if a parent namespace exists.
+        if let Some(namespace) = parent {
+            request_builder = request_builder.query(&[("parent", namespace.to_url_string())]);
+        }
+        let request = request_builder.build()?;
+
+        let handler = |response: Response| async move {
+            match response.status() {
+                StatusCode::OK => {
+                    Ok(deserialize_catalog_response::<ListNamespaceResponse>(response).await?)
+                }
+                StatusCode::NOT_FOUND => Err(Error::new(
+                    ErrorKind::Unexpected,

Review Comment:
   #1038 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "connortsui20 (via GitHub)" <gi...@apache.org>.
connortsui20 commented on code in PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957521755


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -312,90 +317,107 @@ impl RestCatalog {
     }
 }
 
+/// All requests and expected responses are derived from the REST catalog API spec:
+/// https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml
 #[async_trait]
 impl Catalog for RestCatalog {
-    /// List namespaces from table.
     async fn list_namespaces(
         &self,
         parent: Option<&NamespaceIdent>,
     ) -> Result<Vec<NamespaceIdent>> {
-        let mut request = self.context().await?.client.request(
-            Method::GET,
-            self.context().await?.config.namespaces_endpoint(),
-        );
-        if let Some(ns) = parent {
-            request = request.query(&[("parent", ns.to_url_string())]);
-        }
+        let context = self.context().await?;
 
-        let resp = self
-            .context()
-            .await?
+        let mut request_builder = context
             .client
-            .query::<ListNamespaceResponse, ErrorResponse>(request.build()?)
-            .await?;
+            .request(Method::GET, context.config.namespaces_endpoint());
+        // Filter on `parent={namespace}` if a parent namespace exists.
+        if let Some(namespace) = parent {
+            request_builder = request_builder.query(&[("parent", namespace.to_url_string())]);
+        }
+        let request = request_builder.build()?;
+
+        let handler = |response: Response| async move {
+            match response.status() {
+                StatusCode::OK => {
+                    Ok(deserialize_catalog_response::<ListNamespaceResponse>(response).await?)
+                }
+                StatusCode::NOT_FOUND => Err(Error::new(
+                    ErrorKind::Unexpected,

Review Comment:
   I think that if we want an error case for `NamespaceNotExists` there should also be error cases for every other type of failure (for creates, updates, deletes, plus the same for tables). Though I think that should be a different PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#issuecomment-2692076363

   > @Xuanwo could you take a look at the [latest commit](https://github.com/apache/iceberg-rust/pull/965/commits/0f72f01a6963cb97475a2dcd4776aba794ea0932) and see if it makes sense?
   
   Hi, I'm still not in favor of the `handler` style. The stack size of my brain is not sufficient to handle the extra handler (sorry for that). Maybe we could consider the following?
   
   ```rust
   let resp: Response = client.query_catalog(request).await?;
   if response.status() == StatusCode::OK {
       deserialize_catalog_response::<CatalogConfig>(response).await
   } else {
       Err(deserialize_unexpected_catalog_error(response).await)
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "connortsui20 (via GitHub)" <gi...@apache.org>.
connortsui20 commented on code in PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957521755


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -312,90 +317,107 @@ impl RestCatalog {
     }
 }
 
+/// All requests and expected responses are derived from the REST catalog API spec:
+/// https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml
 #[async_trait]
 impl Catalog for RestCatalog {
-    /// List namespaces from table.
     async fn list_namespaces(
         &self,
         parent: Option<&NamespaceIdent>,
     ) -> Result<Vec<NamespaceIdent>> {
-        let mut request = self.context().await?.client.request(
-            Method::GET,
-            self.context().await?.config.namespaces_endpoint(),
-        );
-        if let Some(ns) = parent {
-            request = request.query(&[("parent", ns.to_url_string())]);
-        }
+        let context = self.context().await?;
 
-        let resp = self
-            .context()
-            .await?
+        let mut request_builder = context
             .client
-            .query::<ListNamespaceResponse, ErrorResponse>(request.build()?)
-            .await?;
+            .request(Method::GET, context.config.namespaces_endpoint());
+        // Filter on `parent={namespace}` if a parent namespace exists.
+        if let Some(namespace) = parent {
+            request_builder = request_builder.query(&[("parent", namespace.to_url_string())]);
+        }
+        let request = request_builder.build()?;
+
+        let handler = |response: Response| async move {
+            match response.status() {
+                StatusCode::OK => {
+                    Ok(deserialize_catalog_response::<ListNamespaceResponse>(response).await?)
+                }
+                StatusCode::NOT_FOUND => Err(Error::new(
+                    ErrorKind::Unexpected,

Review Comment:
   I think that if we want an error case for `NamespaceNotExists` there should also be error cases for every other type of failure (for creates, updates, deletes, plus the same for tables)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "connortsui20 (via GitHub)" <gi...@apache.org>.
connortsui20 commented on code in PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957526271


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -507,78 +530,98 @@ impl Catalog for RestCatalog {
             })
             .build()?;
 
-        let resp = self
-            .context()
-            .await?
-            .client
-            .query::<LoadTableResponse, ErrorResponse>(request)
-            .await?;
+        let handler = |response: Response| async move {
+            match response.status() {
+                StatusCode::OK => {
+                    Ok(deserialize_catalog_response::<LoadTableResponse>(response).await?)
+                }
+                StatusCode::NOT_FOUND => Err(Error::new(
+                    ErrorKind::Unexpected,
+                    "Tried to create a table under a namespace that does not exist",
+                )),
+                StatusCode::CONFLICT => Err(Error::new(
+                    ErrorKind::Unexpected,

Review Comment:
   Maybe this should be in a different PR? I'm willing to head a PR to add a sub enum into `ErrorKind` that states things that can go wrong (though that would definitely be a breaking change)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "connortsui20 (via GitHub)" <gi...@apache.org>.
connortsui20 commented on code in PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957528321


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -670,21 +719,47 @@ impl Catalog for RestCatalog {
             })
             .build()?;
 
-        let resp = self
-            .context()
-            .await?
-            .client
-            .query::<CommitTableResponse, ErrorResponse>(request)
-            .await?;
+        let handler = |response: Response| async move {
+            if response.status() == StatusCode::OK {
+                return deserialize_catalog_response::<CommitTableResponse>(response).await;
+            }
+
+            let error_message = match response.status() {
+                StatusCode::NOT_FOUND => "Tried to update a table that does not exist",
+                StatusCode::CONFLICT => {
+                    "CommitFailedException, one or more requirements failed. The client may retry."
+                }
+                StatusCode::INTERNAL_SERVER_ERROR => {
+                    "An unknown server-side problem occurred; the commit state is unknown."
+                }
+                StatusCode::BAD_GATEWAY => {
+                    "A gateway or proxy received an invalid response from the upstream server; the commit state is unknown."
+                }
+                StatusCode::GATEWAY_TIMEOUT => {
+                    "A server-side gateway timeout occurred; the commit state is unknown."
+                }
+                _ => {
+                    return Err(deserialize_unexpected_catalog_error(
+                        response,
+                    )
+                    .await);
+                }
+            };
+
+            Err(Error::new(ErrorKind::Unexpected, error_message))
+        };
+
+        let response = context.client.query_catalog(request, handler).await?;

Review Comment:
   > How about taking a step back and simply returning an HTTP response in those functions? That way, we have full control over handling the HTTP response here. We only have a limited number of APIs to implement, and they won’t grow quickly. I’m fine with just writing them directly instead of adding generics or callbacks.
   
   I can try this, but I'm not sure this would make things cleaner. It just means that you are moving where you call the serialize function from inside the handler to outside the handler, and now then now you have to handle the error cases outside the helper every single time. Maybe as a best of both worlds I can specify the exact type of `response` every time? Like:
   
   ```rust
   let response: ListNamespaceResponse = context.client.query_catalog(request, handler).await?;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "connortsui20 (via GitHub)" <gi...@apache.org>.
connortsui20 commented on PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#issuecomment-2679231638

   @Xuanwo could you take a look at the [latest commit](https://github.com/apache/iceberg-rust/pull/965/commits/e8275d040b9174adde439672246064f4708c8504) and see if it makes sense?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1977531978


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -312,90 +317,107 @@ impl RestCatalog {
     }
 }
 
+/// All requests and expected responses are derived from the REST catalog API spec:
+/// https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml
 #[async_trait]
 impl Catalog for RestCatalog {
-    /// List namespaces from table.
     async fn list_namespaces(
         &self,
         parent: Option<&NamespaceIdent>,
     ) -> Result<Vec<NamespaceIdent>> {
-        let mut request = self.context().await?.client.request(
-            Method::GET,
-            self.context().await?.config.namespaces_endpoint(),
-        );
-        if let Some(ns) = parent {
-            request = request.query(&[("parent", ns.to_url_string())]);
-        }
+        let context = self.context().await?;
 
-        let resp = self
-            .context()
-            .await?
+        let mut request_builder = context
             .client
-            .query::<ListNamespaceResponse, ErrorResponse>(request.build()?)
-            .await?;
+            .request(Method::GET, context.config.namespaces_endpoint());
+        // Filter on `parent={namespace}` if a parent namespace exists.
+        if let Some(namespace) = parent {
+            request_builder = request_builder.query(&[("parent", namespace.to_url_string())]);
+        }
+        let request = request_builder.build()?;
+
+        let handler = |response: Response| async move {
+            match response.status() {
+                StatusCode::OK => {
+                    Ok(deserialize_catalog_response::<ListNamespaceResponse>(response).await?)
+                }
+                StatusCode::NOT_FOUND => Err(Error::new(
+                    ErrorKind::Unexpected,

Review Comment:
   Hi, would you like to create an issue for this? I think we have some errors to track here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "connortsui20 (via GitHub)" <gi...@apache.org>.
connortsui20 commented on PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#issuecomment-2692263199

   > > @Xuanwo could you take a look at the [latest commit](https://github.com/apache/iceberg-rust/pull/965/commits/0f72f01a6963cb97475a2dcd4776aba794ea0932) and see if it makes sense?
   > 
   > Hi, I'm still not in favor of the `handler` style. The stack size of my brain is not sufficient to handle the extra handler (sorry for that). Maybe we could consider the following?
   > 
   > ```rust
   > let resp: Response = client.query_catalog(request).await?;
   > if response.status() == StatusCode::OK {
   >     deserialize_catalog_response::<CatalogConfig>(response).await
   > } else {
   >     Err(deserialize_unexpected_catalog_error(response).await)
   > }
   > ```
   
   Okay, I'll make that change and then we can reevaluate if it makes more sense to delay any serialization errors. You're probably right by the way, I guess I just think about this weirdly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#issuecomment-2703297501

   I'll merge this PR in three days if there are no further objections. I believe it's on the right track, and additional polish can be made on top of it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957506030


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -312,90 +317,107 @@ impl RestCatalog {
     }
 }
 
+/// All requests and expected responses are derived from the REST catalog API spec:
+/// https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml
 #[async_trait]
 impl Catalog for RestCatalog {
-    /// List namespaces from table.
     async fn list_namespaces(
         &self,
         parent: Option<&NamespaceIdent>,
     ) -> Result<Vec<NamespaceIdent>> {
-        let mut request = self.context().await?.client.request(
-            Method::GET,
-            self.context().await?.config.namespaces_endpoint(),
-        );
-        if let Some(ns) = parent {
-            request = request.query(&[("parent", ns.to_url_string())]);
-        }
+        let context = self.context().await?;
 
-        let resp = self
-            .context()
-            .await?
+        let mut request_builder = context
             .client
-            .query::<ListNamespaceResponse, ErrorResponse>(request.build()?)
-            .await?;
+            .request(Method::GET, context.config.namespaces_endpoint());
+        // Filter on `parent={namespace}` if a parent namespace exists.
+        if let Some(namespace) = parent {
+            request_builder = request_builder.query(&[("parent", namespace.to_url_string())]);
+        }
+        let request = request_builder.build()?;
+
+        let handler = |response: Response| async move {
+            match response.status() {
+                StatusCode::OK => {
+                    Ok(deserialize_catalog_response::<ListNamespaceResponse>(response).await?)
+                }
+                StatusCode::NOT_FOUND => Err(Error::new(
+                    ErrorKind::Unexpected,

Review Comment:
   This change made me think about how users can handle cases where the namespace doesn't exist. Maybe we should add an `ErrorKind::NamespaceNotExists`?
   
   Maybe this should be done in another PR.



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -507,78 +530,98 @@ impl Catalog for RestCatalog {
             })
             .build()?;
 
-        let resp = self
-            .context()
-            .await?
-            .client
-            .query::<LoadTableResponse, ErrorResponse>(request)
-            .await?;
+        let handler = |response: Response| async move {
+            match response.status() {
+                StatusCode::OK => {
+                    Ok(deserialize_catalog_response::<LoadTableResponse>(response).await?)
+                }
+                StatusCode::NOT_FOUND => Err(Error::new(
+                    ErrorKind::Unexpected,
+                    "Tried to create a table under a namespace that does not exist",
+                )),
+                StatusCode::CONFLICT => Err(Error::new(
+                    ErrorKind::Unexpected,

Review Comment:
   The same question, maybe we should have `TableNotFound` nad `TableAlreadyExists`.



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -670,21 +719,47 @@ impl Catalog for RestCatalog {
             })
             .build()?;
 
-        let resp = self
-            .context()
-            .await?
-            .client
-            .query::<CommitTableResponse, ErrorResponse>(request)
-            .await?;
+        let handler = |response: Response| async move {
+            if response.status() == StatusCode::OK {
+                return deserialize_catalog_response::<CommitTableResponse>(response).await;
+            }
+
+            let error_message = match response.status() {
+                StatusCode::NOT_FOUND => "Tried to update a table that does not exist",
+                StatusCode::CONFLICT => {
+                    "CommitFailedException, one or more requirements failed. The client may retry."
+                }
+                StatusCode::INTERNAL_SERVER_ERROR => {
+                    "An unknown server-side problem occurred; the commit state is unknown."
+                }
+                StatusCode::BAD_GATEWAY => {
+                    "A gateway or proxy received an invalid response from the upstream server; the commit state is unknown."
+                }
+                StatusCode::GATEWAY_TIMEOUT => {
+                    "A server-side gateway timeout occurred; the commit state is unknown."
+                }
+                _ => {
+                    return Err(deserialize_unexpected_catalog_error(
+                        response,
+                    )
+                    .await);
+                }
+            };
+
+            Err(Error::new(ErrorKind::Unexpected, error_message))
+        };
+
+        let response = context.client.query_catalog(request, handler).await?;

Review Comment:
   To be honest, I don't like the style of creating a handle and passing it into another function. It usually breaks the logical flow and makes it hard to understand what’s happening.
   
   It's easy to understand that:
   
   ```rust
   let resp = self
           .context()
           .await?
           .client
           .query::<CommitTableResponse, ErrorResponse>(request)
           .await?;
   ```
   
   Ok, the client will perform query and returns a response or error.
   
   But it's hard to:
   
   ```rust
   let handler = |response: Response| async move { ... };
   let response = context.client.query_catalog(request, handler).await?;
   ```
   
   Let's build a handler and it will do something to the response. Remember this funcation now. The client will perform query catalog and use the previous handler to parse the response. Emmmm, wait a second, what does this handler do?
   
   ---
   
   How about taking a step back and simply returning an HTTP response in those functions? That way, we have full control over handling the HTTP response here. We only have a limited number of APIs to implement, and they won’t grow quickly. I’m fine with just writing them directly instead of adding generics or callbacks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "connortsui20 (via GitHub)" <gi...@apache.org>.
connortsui20 commented on code in PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957524251


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -670,21 +719,47 @@ impl Catalog for RestCatalog {
             })
             .build()?;
 
-        let resp = self
-            .context()
-            .await?
-            .client
-            .query::<CommitTableResponse, ErrorResponse>(request)
-            .await?;
+        let handler = |response: Response| async move {
+            if response.status() == StatusCode::OK {
+                return deserialize_catalog_response::<CommitTableResponse>(response).await;
+            }
+
+            let error_message = match response.status() {
+                StatusCode::NOT_FOUND => "Tried to update a table that does not exist",
+                StatusCode::CONFLICT => {
+                    "CommitFailedException, one or more requirements failed. The client may retry."
+                }
+                StatusCode::INTERNAL_SERVER_ERROR => {
+                    "An unknown server-side problem occurred; the commit state is unknown."
+                }
+                StatusCode::BAD_GATEWAY => {
+                    "A gateway or proxy received an invalid response from the upstream server; the commit state is unknown."
+                }
+                StatusCode::GATEWAY_TIMEOUT => {
+                    "A server-side gateway timeout occurred; the commit state is unknown."
+                }
+                _ => {
+                    return Err(deserialize_unexpected_catalog_error(
+                        response,
+                    )
+                    .await);
+                }
+            };
+
+            Err(Error::new(ErrorKind::Unexpected, error_message))
+        };
+
+        let response = context.client.query_catalog(request, handler).await?;

Review Comment:
   > Ok, the client will perform query and returns a response or error.
   
   I agree that this is clearer, but the whole point of this PR is that by having a single `query` function you are unable to handle each API call's response specifically. There will be cases when only allowing `OK` or `NO_CONTENT` is not enough: for example in the `loadTable` response it should be able to handle `StatusCode::NOT_MODIFIED` (304) as a correct case. There should have to be a new kind of `query` or `execute` function for every case that needs to be handled.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "connortsui20 (via GitHub)" <gi...@apache.org>.
connortsui20 commented on code in PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957528321


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -670,21 +719,47 @@ impl Catalog for RestCatalog {
             })
             .build()?;
 
-        let resp = self
-            .context()
-            .await?
-            .client
-            .query::<CommitTableResponse, ErrorResponse>(request)
-            .await?;
+        let handler = |response: Response| async move {
+            if response.status() == StatusCode::OK {
+                return deserialize_catalog_response::<CommitTableResponse>(response).await;
+            }
+
+            let error_message = match response.status() {
+                StatusCode::NOT_FOUND => "Tried to update a table that does not exist",
+                StatusCode::CONFLICT => {
+                    "CommitFailedException, one or more requirements failed. The client may retry."
+                }
+                StatusCode::INTERNAL_SERVER_ERROR => {
+                    "An unknown server-side problem occurred; the commit state is unknown."
+                }
+                StatusCode::BAD_GATEWAY => {
+                    "A gateway or proxy received an invalid response from the upstream server; the commit state is unknown."
+                }
+                StatusCode::GATEWAY_TIMEOUT => {
+                    "A server-side gateway timeout occurred; the commit state is unknown."
+                }
+                _ => {
+                    return Err(deserialize_unexpected_catalog_error(
+                        response,
+                    )
+                    .await);
+                }
+            };
+
+            Err(Error::new(ErrorKind::Unexpected, error_message))
+        };
+
+        let response = context.client.query_catalog(request, handler).await?;

Review Comment:
   > How about taking a step back and simply returning an HTTP response in those functions? That way, we have full control over handling the HTTP response here. We only have a limited number of APIs to implement, and they won’t grow quickly. I’m fine with just writing them directly instead of adding generics or callbacks.
   
   I can try this, but I'm not sure this would make things cleaner. It just means that you are moving where you call the serialize function from inside the handler to outside the handler, and now then now you have to handle the error cases outside the helper every single time. Maybe as a best of both worlds I can specify the exact type of `response` every time? Like:
   
   ```rust
   let response: ListNamespaceResponse = context.client.query_catalog(request, handler).await?;
   ```
   
   I could also rename `response` to `deserialized_response` to make it super obvious what is happening.
   
   ---
   
   Edit: After thinking a bit more about this, I'm trying to figure out what the signature of the custom handler should be. If it is not an asynchronous `FnOnce(Response) -> Result<R>`, then should it be a `FnOnce(Response) -> Result<Response>`? So the handler takes in the HTTP response and checks if there are any problems with it. If there are problems, then it returns `Err`. But if it can't detect any problems, it returns the entire `Response` even though it is still possible that the deserialization goes wrong? I think this could be more confusing, but I can see points on both sides.
   
   I should also mention that the `deserialize_unexpected_catalog_error` should also be deserializing every response it gets into a proper error message instead of a blanket "Received unexpected response", but I felt that was out of scope for this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

Posted by "connortsui20 (via GitHub)" <gi...@apache.org>.
connortsui20 commented on code in PR #965:
URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957528321


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -670,21 +719,47 @@ impl Catalog for RestCatalog {
             })
             .build()?;
 
-        let resp = self
-            .context()
-            .await?
-            .client
-            .query::<CommitTableResponse, ErrorResponse>(request)
-            .await?;
+        let handler = |response: Response| async move {
+            if response.status() == StatusCode::OK {
+                return deserialize_catalog_response::<CommitTableResponse>(response).await;
+            }
+
+            let error_message = match response.status() {
+                StatusCode::NOT_FOUND => "Tried to update a table that does not exist",
+                StatusCode::CONFLICT => {
+                    "CommitFailedException, one or more requirements failed. The client may retry."
+                }
+                StatusCode::INTERNAL_SERVER_ERROR => {
+                    "An unknown server-side problem occurred; the commit state is unknown."
+                }
+                StatusCode::BAD_GATEWAY => {
+                    "A gateway or proxy received an invalid response from the upstream server; the commit state is unknown."
+                }
+                StatusCode::GATEWAY_TIMEOUT => {
+                    "A server-side gateway timeout occurred; the commit state is unknown."
+                }
+                _ => {
+                    return Err(deserialize_unexpected_catalog_error(
+                        response,
+                    )
+                    .await);
+                }
+            };
+
+            Err(Error::new(ErrorKind::Unexpected, error_message))
+        };
+
+        let response = context.client.query_catalog(request, handler).await?;

Review Comment:
   > How about taking a step back and simply returning an HTTP response in those functions? That way, we have full control over handling the HTTP response here. We only have a limited number of APIs to implement, and they won’t grow quickly. I’m fine with just writing them directly instead of adding generics or callbacks.
   
   I can try this, but I'm not sure this would make things cleaner. It just means that you are moving where you call the serialize function from inside the handler to outside the handler, and now then now you have to handle the error cases outside the helper every single time. Maybe as a best of both worlds I can specify the exact type of `response` every time? Like:
   
   ```rust
   let response: ListNamespaceResponse = context.client.query_catalog(request, handler).await?;
   ```
   
   I could also rename `response` to `deserialized_response` to make it super obvious what is happening.
   
   ---
   
   Edit: After thinking a bit more about this, I'm trying to figure out what the signature of the custom handler should be. If it is not an asynchronous `FnOnce(Response) -> Result<R>`, then should it be a `FnOnce(Response) -> Result<Response>`? So the handler takes in the HTTP response and checks if there are any problems with it. If there are problems, then it returns `Err`. But if it can't detect any problems, it returns the entire `Response` even though it is still possible that the deserialization goes wrong? I think this could be more confusing, but I can see points on both sides.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org