You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "TennyZhuang (via GitHub)" <gi...@apache.org> on 2024/03/11 14:18:28 UTC

[PR] feat: implement OAuth for catalog rest client [iceberg-rust]

TennyZhuang opened a new pull request, #254:
URL: https://github.com/apache/iceberg-rust/pull/254

   Resolve #238 
   
   The docker image of tabular doesn't support credential settings, so it's impossible to write a test.
   
   I just test it manually using their hosted iceberg service.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1521025132


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -113,6 +117,19 @@ impl RestCatalogConfig {
             ),
         ]);
 
+        if let Some(token) = self.props.get("token") {

Review Comment:
   I just follow the previous style in the rest client implementation. If we should extract a constant, we should do it in a separate PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1521041665


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }
+        if let Some(credential) = self.config.props.get("credential") {
+            let (client_id, client_secret) = if credential.contains(':') {
+                let (client_id, client_secret) = credential.split_once(':').unwrap();
+                (Some(client_id), client_secret)
+            } else {
+                (None, credential.as_str())
+            };
+            let mut params = HashMap::with_capacity(4);
+            params.insert("grant_type", "client_credentials");
+            if let Some(client_id) = client_id {
+                params.insert("client_id", client_id);
+            }
+            params.insert("client_secret", client_secret);
+            params.insert("scope", "catalog");
+            let req = self
+                .client
+                .0
+                .post(self.config.get_token_endpoint())

Review Comment:
   Make sense, will add it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1521174970


##########
crates/catalog/rest/tests/rest_catalog_test.rs:
##########


Review Comment:
   Could you add some tests here to mock fetch token process?



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -113,6 +117,19 @@ impl RestCatalogConfig {
             ),
         ]);
 
+        if let Some(token) = self.props.get("token") {

Review Comment:
   Fair enough.



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;

Review Comment:
   Sorry, I don't get your point. I know that we should recreate http client, what I mean is that is it required that `update_config` happens before `fetch_access_token`? Seems java does this in reverse order, so I guess it doesn't matter.



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }

Review Comment:
   Sorry, I still don't get your point. But currently we in fact don't refresh tokens, so this logic is same.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1521635258


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;

Review Comment:
   > what I mean is that is it required that update_config happens before fetch_access_token?
   
   `fetch_access_token` happens before `update_config` in my code, and also in the python code.
   
   Some config API also need authentication, such as `https://api.tabular.io/ws/v1/config`



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -144,13 +161,15 @@ impl HttpClient {
                 .with_source(e)
             })?)
         } else {
+            let code = resp.status();
             let text = resp.bytes().await?;
             let e = serde_json::from_slice::<E>(&text).map_err(|e| {
                 Error::new(
                     ErrorKind::Unexpected,
                     "Failed to parse response from rest catalog server!",
                 )
                 .with_context("json", String::from_utf8_lossy(&text))
+                .with_context("code", code.to_string())

Review Comment:
   `execute` already had that. It's inconsistent before the PR.



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }

Review Comment:
   Anyway, the logic is same as a specified python version, I think it's acceptable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1521025132


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -113,6 +117,19 @@ impl RestCatalogConfig {
             ),
         ]);
 
+        if let Some(token) = self.props.get("token") {

Review Comment:
   I just follow the previous style. If we should extract a constant, we should do it in a separate PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1520864738


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }
+        if let Some(credential) = self.config.props.get("credential") {
+            let (client_id, client_secret) = if credential.contains(':') {
+                let (client_id, client_secret) = credential.split_once(':').unwrap();
+                (Some(client_id), client_secret)
+            } else {
+                (None, credential.as_str())
+            };
+            let mut params = HashMap::with_capacity(4);
+            params.insert("grant_type", "client_credentials");
+            if let Some(client_id) = client_id {
+                params.insert("client_id", client_id);
+            }
+            params.insert("client_secret", client_secret);
+            params.insert("scope", "catalog");

Review Comment:
   I've already made `issued_token_type` optional. For configurable `scope`, I guess I can do it in later PR? or @himadripal can help that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1521046528


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;

Review Comment:
   It's necessary. For example, in the tabular's hosted iceberg, they will return a new `uri` to override your config. It's necessary to recreate a new http client.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "himadripal (via GitHub)" <gi...@apache.org>.
himadripal commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1520307628


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }
+        if let Some(credential) = self.config.props.get("credential") {
+            let (client_id, client_secret) = if credential.contains(':') {
+                let (client_id, client_secret) = credential.split_once(':').unwrap();
+                (Some(client_id), client_secret)
+            } else {
+                (None, credential.as_str())
+            };
+            let mut params = HashMap::with_capacity(4);
+            params.insert("grant_type", "client_credentials");
+            if let Some(client_id) = client_id {
+                params.insert("client_id", client_id);
+            }
+            params.insert("client_secret", client_secret);
+            params.insert("scope", "catalog");

Review Comment:
   @TennyZhuang I was doing a rust PR to port the changes from python and java PR mentioned @flyrain mentioned. Apart from making the scope and those additional param configurable, in java/python version, there is an exchange flow as well - when the context has a token and using that getting a new token and to refresh token before expiry.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#issuecomment-1990529796

   > Is there any way to extend the Dockerfile with the Env for credentials?
   
   I can't find any codes about OAuth in the image. Only an open issue seems related.
   
   https://github.com/tabular-io/iceberg-rest-image/issues/27


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "himadripal (via GitHub)" <gi...@apache.org>.
himadripal commented on PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#issuecomment-1999041701

   @TennyZhuang @flyrain  once this is merged, I can make the changes for scope, and other optional param in a followup PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#issuecomment-1991227015

   > In that version you referred, token refresh is not implemented, I prefer to add it in later PR, what do you think?
   
   Sounds reasonable to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#issuecomment-1990934283

   BTW, is it possible to update [examples](https://github.com/apache/iceberg-rust/blob/be65c89001307d56e1c7f2ca000ef17e026e7c23/crates/examples/src/rest_catalog_namespace.rs) so that users could try in?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#issuecomment-1999373096

   I think the PR is ready for review as a minimal implementation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1521040362


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {

Review Comment:
   https://github.com/apache/iceberg-python/blob/b026c5d4b7ba5244f67733870c06c952923e6852/pyiceberg/catalog/rest.py#L268
   
   Will rename it later when it really refresh.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1524722783


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }
+        if let Some(credential) = self.config.props.get("credential") {
+            let (client_id, client_secret) = if credential.contains(':') {
+                let (client_id, client_secret) = credential.split_once(':').unwrap();
+                (Some(client_id), client_secret)
+            } else {
+                (None, credential.as_str())
+            };
+            let mut params = HashMap::with_capacity(4);
+            params.insert("grant_type", "client_credentials");
+            if let Some(client_id) = client_id {
+                params.insert("client_id", client_id);
+            }
+            params.insert("client_secret", client_secret);
+            params.insert("scope", "catalog");
+            let req = self
+                .client
+                .0
+                .post(self.config.get_token_endpoint())

Review Comment:
   https://github.com/apache/iceberg-python/blob/b026c5d4b7ba5244f67733870c06c952923e6852/pyiceberg/catalog/rest.py#L268
   
   Refer to the version, will add in later PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1524725484


##########
crates/catalog/rest/tests/rest_catalog_test.rs:
##########


Review Comment:
   Added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#issuecomment-1991016398

   > Hi, @TennyZhuang Thanks for this pr! It looks great, but there are something missing:
   > 
   > 1. We should try to refresh token when error happens
   > 2. Please make the property names constants.
   
   Hi, the implementation is mainly copied from an old python implementation about 5 months ago, which is referenced by your issue. I think the most naive implementation can make the PR easier for review.
   
   In that version you referred, token refresh is not implemented.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "himadripal (via GitHub)" <gi...@apache.org>.
himadripal commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1520307628


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }
+        if let Some(credential) = self.config.props.get("credential") {
+            let (client_id, client_secret) = if credential.contains(':') {
+                let (client_id, client_secret) = credential.split_once(':').unwrap();
+                (Some(client_id), client_secret)
+            } else {
+                (None, credential.as_str())
+            };
+            let mut params = HashMap::with_capacity(4);
+            params.insert("grant_type", "client_credentials");
+            if let Some(client_id) = client_id {
+                params.insert("client_id", client_id);
+            }
+            params.insert("client_secret", client_secret);
+            params.insert("scope", "catalog");

Review Comment:
   @TennyZhuang I was doing a rust PR to port the changes from python and java PR mentioned @flyrain mentioned. Apart from making the scope and those additional param configurable, in java/python version, there is an exchange flow 
   - when the context has a token and using that getting a new token
   -  to refresh token before expiry.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#issuecomment-1997291580

   > BTW, is it possible to update [examples](https://github.com/apache/iceberg-rust/blob/be65c89001307d56e1c7f2ca000ef17e026e7c23/crates/examples/src/rest_catalog_namespace.rs) so that users could try in?
   
   Will do that after support refresh token.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "marvinlanhenke (via GitHub)" <gi...@apache.org>.
marvinlanhenke commented on PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#issuecomment-1990440706

   > Resolve #238 
   > 
   > The docker image of tabular doesn't support credential settings, so it's impossible to write a test.
   > 
   > I just test it manually using their hosted iceberg service.
   
   Is there any way to extend the Dockerfile with the Env for credentials? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1521027528


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }

Review Comment:
   It's necessary, since we recreate the client after calling `update_config`. It can be avoid when we do some refactors.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "himadripal (via GitHub)" <gi...@apache.org>.
himadripal commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1520307628


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }
+        if let Some(credential) = self.config.props.get("credential") {
+            let (client_id, client_secret) = if credential.contains(':') {
+                let (client_id, client_secret) = credential.split_once(':').unwrap();
+                (Some(client_id), client_secret)
+            } else {
+                (None, credential.as_str())
+            };
+            let mut params = HashMap::with_capacity(4);
+            params.insert("grant_type", "client_credentials");
+            if let Some(client_id) = client_id {
+                params.insert("client_id", client_id);
+            }
+            params.insert("client_secret", client_secret);
+            params.insert("scope", "catalog");

Review Comment:
   @TennyZhuang I was doing a rust PR to port the changes from python and java PR mentioned @flyrain mentioned. Apart from making the scope and those additional param configurable, in java/python version, there is an exchange flow - when the context has a token and using that getting a new token and to refresh token before expiry.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1520950487


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -113,6 +117,19 @@ impl RestCatalogConfig {
             ),
         ]);
 
+        if let Some(token) = self.props.get("token") {
+            headers.insert(
+                "Authorization",

Review Comment:
   Ditto.



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -113,6 +117,19 @@ impl RestCatalogConfig {
             ),
         ]);
 
+        if let Some(token) = self.props.get("token") {

Review Comment:
   Please make this a constant.



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {

Review Comment:
   ```suggestion
       async fn refresh_access_token(&mut self) -> Result<()> {
   ```



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -144,13 +161,15 @@ impl HttpClient {
                 .with_source(e)
             })?)
         } else {
+            let code = resp.status();
             let text = resp.bytes().await?;
             let e = serde_json::from_slice::<E>(&text).map_err(|e| {
                 Error::new(
                     ErrorKind::Unexpected,
                     "Failed to parse response from rest catalog server!",
                 )
                 .with_context("json", String::from_utf8_lossy(&text))
+                .with_context("code", code.to_string())

Review Comment:
   Also add it for `execute`?



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }
+        if let Some(credential) = self.config.props.get("credential") {
+            let (client_id, client_secret) = if credential.contains(':') {
+                let (client_id, client_secret) = credential.split_once(':').unwrap();
+                (Some(client_id), client_secret)
+            } else {
+                (None, credential.as_str())
+            };
+            let mut params = HashMap::with_capacity(4);
+            params.insert("grant_type", "client_credentials");
+            if let Some(client_id) = client_id {
+                params.insert("client_id", client_id);
+            }
+            params.insert("client_secret", client_secret);
+            params.insert("scope", "catalog");
+            let req = self
+                .client
+                .0
+                .post(self.config.get_token_endpoint())

Review Comment:
   In fact the auth url could be configurable: https://github.com/apache/iceberg-python/blob/70342ac83d2d1f121f3ab04c6d7317c8830fdad1/pyiceberg/catalog/rest.py#L306



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }

Review Comment:
   I don't think we should add this check since toke may expire, and we need to refresh it.



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;

Review Comment:
   The python implementation update config first, then recreate client? Is this required or we can relax? cc @Fokko @flyrain 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 merged PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1520276689


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -626,6 +688,14 @@ mod _serde {
         }
     }
 
+    #[derive(Debug, Serialize, Deserialize)]
+    pub(super) struct TokenResponse {
+        pub(super) access_token: String,
+        pub(super) token_type: String,
+        pub(super) expires_in: u64,
+        pub(super) issued_token_type: String,

Review Comment:
   `issued_token_type` should be optional, here is a python fix for reference, https://github.com/apache/iceberg-python/pull/466



##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }
+        if let Some(credential) = self.config.props.get("credential") {
+            let (client_id, client_secret) = if credential.contains(':') {
+                let (client_id, client_secret) = credential.split_once(':').unwrap();
+                (Some(client_id), client_secret)
+            } else {
+                (None, credential.as_str())
+            };
+            let mut params = HashMap::with_capacity(4);
+            params.insert("grant_type", "client_credentials");
+            if let Some(client_id) = client_id {
+                params.insert("client_id", client_id);
+            }
+            params.insert("client_secret", client_secret);
+            params.insert("scope", "catalog");

Review Comment:
   `Scope` should be configurable, and we will also need optional configurable `audience` and `resource`. Here is the python PR for reference, https://github.com/apache/iceberg-python/pull/486. 
   
   I'd also recommend to group required fields and optional fields, so that the request is more extendable, which makes it easy to add new parameters in the future. Here are related discussion, https://github.com/apache/iceberg/pull/9839#discussion_r1511897858
   cc @himadripal



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "himadripal (via GitHub)" <gi...@apache.org>.
himadripal commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1520307628


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -497,13 +516,56 @@ impl RestCatalog {
             client: config.try_create_rest_client()?,
             config,
         };
-
+        catalog.fetch_access_token().await?;
+        catalog.client = catalog.config.try_create_rest_client()?;
         catalog.update_config().await?;
         catalog.client = catalog.config.try_create_rest_client()?;
 
         Ok(catalog)
     }
 
+    async fn fetch_access_token(&mut self) -> Result<()> {
+        if self.config.props.contains_key("token") {
+            return Ok(());
+        }
+        if let Some(credential) = self.config.props.get("credential") {
+            let (client_id, client_secret) = if credential.contains(':') {
+                let (client_id, client_secret) = credential.split_once(':').unwrap();
+                (Some(client_id), client_secret)
+            } else {
+                (None, credential.as_str())
+            };
+            let mut params = HashMap::with_capacity(4);
+            params.insert("grant_type", "client_credentials");
+            if let Some(client_id) = client_id {
+                params.insert("client_id", client_id);
+            }
+            params.insert("client_secret", client_secret);
+            params.insert("scope", "catalog");

Review Comment:
   @TennyZhuang I was doing a rust PR to port the changes from python and java PR mentioned above by @flyrain. Apart from making the scope and those additional param configurable, in java/python version, there is an exchange flow 
   - when the context has a token and using that getting a new token
   -  to refresh token before expiry.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #254:
URL: https://github.com/apache/iceberg-rust/pull/254#discussion_r1521837108


##########
crates/catalog/rest/src/catalog.rs:
##########
@@ -113,6 +117,19 @@ impl RestCatalogConfig {
             ),
         ]);
 
+        if let Some(token) = self.props.get("token") {
+            headers.insert(
+                "Authorization",

Review Comment:
   Can we use the const from the http lib? for example, `header::AUTHORIZATION`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org