You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "010Soham (via GitHub)" <gi...@apache.org> on 2025/12/19 11:50:43 UTC

[PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

010Soham opened a new pull request, #2846:
URL: https://github.com/apache/iceberg-python/pull/2846

   
   What does this change do?
   - RestCatalog now passes its AuthManager into FileIO so downstream components can reuse a live token.
   - S3V4RestSigner now calls the AuthManager’s auth_header() when no static token is provided, ensuring the signer gets a fresh bearer token.
   - Added a unit test to verify the signer pulls the Authorization header from an AuthManager.
   
   Why is this needed?
   - After the AuthManager refactor, the signer no longer received a token, causing remote signing to 401 for REST catalog users (e.g., Lakekeeper/MinIO). This restores token propagation and refresh.
   
   How was this tested?
   - make lint
   - make test
   - uv run python -m pytest tests/io/test_fsspec.py -k auth_manager -v
   
   Closes #2544


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2646150788


##########
pyiceberg/io/fsspec.py:
##########
@@ -121,9 +121,19 @@ def __call__(self, request: "AWSRequest", **_: Any) -> None:
         signer_url = self.properties.get(S3_SIGNER_URI, self.properties[URI]).rstrip("/")  # type: ignore
         signer_endpoint = self.properties.get(S3_SIGNER_ENDPOINT, S3_SIGNER_ENDPOINT_DEFAULT)
 
-        signer_headers = {}
+        signer_headers: dict[str, str] = {}
+
+        auth_header: str | None = None
         if token := self.properties.get(TOKEN):
-            signer_headers = {"Authorization": f"Bearer {token}"}
+            auth_header = f"Bearer {token}"

Review Comment:
   yea great point. i think it would be better to split out the "rest signer" from fileio. there's a good example already in the REST catalog, https://github.com/apache/iceberg-python/blob/a99dcada431a06329959cb7f71bf961343256bb0/pyiceberg/catalog/rest/__init__.py#L409-L459
   
   it might also be easier to just pass in the request Session from the REST catalog to the Signer. So we dont need to recreate the auth header directly
   
   but again, we can refactor this after the bug fix :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "martyngigg (via GitHub)" <gi...@apache.org>.
martyngigg commented on PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#issuecomment-3709600864

   I can also confirm that the solution works for me too. Thanks for the fix!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2660924220


##########
pyiceberg/catalog/rest/__init__.py:
##########
@@ -364,16 +359,24 @@ def _create_session(self) -> Session:
             if auth_type != CUSTOM and auth_impl:
                 raise ValueError("auth.impl can only be specified when using custom auth.type")
 
-            session.auth = AuthManagerAdapter(AuthManagerFactory.create(auth_impl or auth_type, auth_type_config))
+            self._auth_manager = AuthManagerFactory.create(auth_impl or auth_type, auth_type_config)
+            session.auth = AuthManagerAdapter(self._auth_manager)
         else:
-            session.auth = AuthManagerAdapter(self._create_legacy_oauth2_auth_manager(session))
+            self._auth_manager = self._create_legacy_oauth2_auth_manager(session)
+            session.auth = AuthManagerAdapter(self._auth_manager)
 
         # Configure SigV4 Request Signing
         if property_as_bool(self.properties, SIGV4, False):
             self._init_sigv4(session)
 
         return session
 
+    def _load_file_io(self, properties: Properties = EMPTY_DICT, location: str | None = None) -> FileIO:
+        merged_properties = {**self.properties, **properties}
+        if self._auth_manager:
+            merged_properties[AUTH_MANAGER] = self._auth_manager
+        return load_file_io(merged_properties, location)
+

Review Comment:
   This method doesn't seem to be used?
   ```suggestion
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2646410874


##########
pyiceberg/io/fsspec.py:
##########
@@ -121,9 +121,19 @@ def __call__(self, request: "AWSRequest", **_: Any) -> None:
         signer_url = self.properties.get(S3_SIGNER_URI, self.properties[URI]).rstrip("/")  # type: ignore
         signer_endpoint = self.properties.get(S3_SIGNER_ENDPOINT, S3_SIGNER_ENDPOINT_DEFAULT)
 
-        signer_headers = {}
+        signer_headers: dict[str, str] = {}
+
+        auth_header: str | None = None
         if token := self.properties.get(TOKEN):
-            signer_headers = {"Authorization": f"Bearer {token}"}

Review Comment:
   I think this was partially mentioned but, should the auth manager take precedence over token when both are set? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#issuecomment-3764261965

   Thanks for the contribution @010Soham and thank you @sungwy @Fokko @geruh for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2645920570


##########
pyiceberg/io/fsspec.py:
##########
@@ -121,9 +121,19 @@ def __call__(self, request: "AWSRequest", **_: Any) -> None:
         signer_url = self.properties.get(S3_SIGNER_URI, self.properties[URI]).rstrip("/")  # type: ignore
         signer_endpoint = self.properties.get(S3_SIGNER_ENDPOINT, S3_SIGNER_ENDPOINT_DEFAULT)
 
-        signer_headers = {}
+        signer_headers: dict[str, str] = {}
+
+        auth_header: str | None = None
         if token := self.properties.get(TOKEN):
-            signer_headers = {"Authorization": f"Bearer {token}"}
+            auth_header = f"Bearer {token}"

Review Comment:
   This may be out of scope for this bug-fix PR, but it looks like we’re tightly coupling `AuthManager` and authentication tokens which are `RestCatalog` concepts into `FileIO`, which should be `Catalog` type agnostic.
   
   It might be worth revisiting this design in more detail in the future to ensure we don’t introduce fallback logic that’s driven by configuration properties rather than clearer separation of concerns



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2646166382


##########
pyiceberg/io/fsspec.py:
##########
@@ -121,9 +121,19 @@ def __call__(self, request: "AWSRequest", **_: Any) -> None:
         signer_url = self.properties.get(S3_SIGNER_URI, self.properties[URI]).rstrip("/")  # type: ignore
         signer_endpoint = self.properties.get(S3_SIGNER_ENDPOINT, S3_SIGNER_ENDPOINT_DEFAULT)
 
-        signer_headers = {}
+        signer_headers: dict[str, str] = {}
+
+        auth_header: str | None = None
         if token := self.properties.get(TOKEN):
-            signer_headers = {"Authorization": f"Bearer {token}"}
+            auth_header = f"Bearer {token}"

Review Comment:
   agreed - just leaving a comment so we don't forget 🙂 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#issuecomment-3688446991

   i was able to verify this locally by using the amazing repro script from @martyngigg
   See https://github.com/apache/iceberg-python/issues/2544#issuecomment-3688446424


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#issuecomment-3757534546

   leaving this open for a few more reviews, i plan to merge this first thing tmr 😄 
   super exciting, we'll be able to release 0.11 soon after this fix 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#issuecomment-3682731155

   @c-thiel Are you able to check if this fixes the issue with the server signed urls?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "fboerman (via GitHub)" <gi...@apache.org>.
fboerman commented on PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#issuecomment-3705502310

   I ran into the issue linked and can confirm using this PR with pyiceberg[s3fs,rest-sigv4] @ git+https://github.com/apache/iceberg-python@refs/pull/2846/head works. would be great to get this merged


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "010Soham (via GitHub)" <gi...@apache.org>.
010Soham commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2663928925


##########
pyiceberg/io/fsspec.py:
##########
@@ -121,9 +121,19 @@ def __call__(self, request: "AWSRequest", **_: Any) -> None:
         signer_url = self.properties.get(S3_SIGNER_URI, self.properties[URI]).rstrip("/")  # type: ignore
         signer_endpoint = self.properties.get(S3_SIGNER_ENDPOINT, S3_SIGNER_ENDPOINT_DEFAULT)
 
-        signer_headers = {}
+        signer_headers: dict[str, str] = {}
+
+        auth_header: str | None = None
         if token := self.properties.get(TOKEN):
-            signer_headers = {"Authorization": f"Bearer {token}"}

Review Comment:
   updated the code so that AuthManager now takes precedence over token when both are set 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "010Soham (via GitHub)" <gi...@apache.org>.
010Soham commented on PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#issuecomment-3713511706

   Thanks everyone for your suggestions and testing things out
   @Fokko 
   I went through all the pending comments and addressed them:
   
   - S3V4RestSigner now prefers auth_manager.auth_header() (called directly) and falls back to token only if no auth manager is present.
   - Moved AUTH_MANAGER to auth.py and updated imports.
   
   All CI checks are green now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2696434765


##########
pyiceberg/catalog/rest/__init__.py:
##########
@@ -364,16 +359,24 @@ def _create_session(self) -> Session:
             if auth_type != CUSTOM and auth_impl:
                 raise ValueError("auth.impl can only be specified when using custom auth.type")
 
-            session.auth = AuthManagerAdapter(AuthManagerFactory.create(auth_impl or auth_type, auth_type_config))
+            self._auth_manager = AuthManagerFactory.create(auth_impl or auth_type, auth_type_config)
+            session.auth = AuthManagerAdapter(self._auth_manager)
         else:
-            session.auth = AuthManagerAdapter(self._create_legacy_oauth2_auth_manager(session))
+            self._auth_manager = self._create_legacy_oauth2_auth_manager(session)
+            session.auth = AuthManagerAdapter(self._auth_manager)

Review Comment:
   nit: take `session.auth = AuthManagerAdapter(self._auth_manager)` part out of the if/else branch since its the same logic



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "010Soham (via GitHub)" <gi...@apache.org>.
010Soham commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2663953951


##########
pyiceberg/catalog/rest/__init__.py:
##########
@@ -364,16 +359,24 @@ def _create_session(self) -> Session:
             if auth_type != CUSTOM and auth_impl:
                 raise ValueError("auth.impl can only be specified when using custom auth.type")
 
-            session.auth = AuthManagerAdapter(AuthManagerFactory.create(auth_impl or auth_type, auth_type_config))
+            self._auth_manager = AuthManagerFactory.create(auth_impl or auth_type, auth_type_config)
+            session.auth = AuthManagerAdapter(self._auth_manager)
         else:
-            session.auth = AuthManagerAdapter(self._create_legacy_oauth2_auth_manager(session))
+            self._auth_manager = self._create_legacy_oauth2_auth_manager(session)
+            session.auth = AuthManagerAdapter(self._auth_manager)
 
         # Configure SigV4 Request Signing
         if property_as_bool(self.properties, SIGV4, False):
             self._init_sigv4(session)
 
         return session
 
+    def _load_file_io(self, properties: Properties = EMPTY_DICT, location: str | None = None) -> FileIO:
+        merged_properties = {**self.properties, **properties}
+        if self._auth_manager:
+            merged_properties[AUTH_MANAGER] = self._auth_manager
+        return load_file_io(merged_properties, location)
+

Review Comment:
   We are using it when the REST catalog builds tables. For example, _response_to_table and _response_to_staged_table call self._load_file_io(...) to construct the Table’s FileIO.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "010Soham (via GitHub)" <gi...@apache.org>.
010Soham commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2663926934


##########
pyiceberg/catalog/__init__.py:
##########
@@ -77,6 +77,7 @@
 _ENV_CONFIG = Config()
 
 TOKEN = "token"
+AUTH_MANAGER = "auth.manager"

Review Comment:
   moved AUTH_MANAGER into auth.py with the other auth code and updated the imports accordingly 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2644611592


##########
pyiceberg/io/fsspec.py:
##########
@@ -121,9 +121,19 @@ def __call__(self, request: "AWSRequest", **_: Any) -> None:
         signer_url = self.properties.get(S3_SIGNER_URI, self.properties[URI]).rstrip("/")  # type: ignore
         signer_endpoint = self.properties.get(S3_SIGNER_ENDPOINT, S3_SIGNER_ENDPOINT_DEFAULT)
 
-        signer_headers = {}
+        signer_headers: dict[str, str] = {}
+
+        auth_header: str | None = None
         if token := self.properties.get(TOKEN):
-            signer_headers = {"Authorization": f"Bearer {token}"}
+            auth_header = f"Bearer {token}"

Review Comment:
   nit: maybe we can get rid of accessing `TOKEN` through properties here and just standardize on using the auth manager. 



##########
pyiceberg/io/fsspec.py:
##########
@@ -121,9 +121,19 @@ def __call__(self, request: "AWSRequest", **_: Any) -> None:
         signer_url = self.properties.get(S3_SIGNER_URI, self.properties[URI]).rstrip("/")  # type: ignore
         signer_endpoint = self.properties.get(S3_SIGNER_ENDPOINT, S3_SIGNER_ENDPOINT_DEFAULT)
 
-        signer_headers = {}
+        signer_headers: dict[str, str] = {}
+
+        auth_header: str | None = None
         if token := self.properties.get(TOKEN):
-            signer_headers = {"Authorization": f"Bearer {token}"}
+            auth_header = f"Bearer {token}"
+        elif auth_manager := self.properties.get(AUTH_MANAGER):
+            header = getattr(auth_manager, "auth_header", None)
+            if callable(header):
+                auth_header = header()

Review Comment:
   ```suggestion
               auth_header = auth_manager.auth_header()
   ```
   could we just call the function directly? this will fail if the `auth_header` function does not exist. 
   i think the current solution will fail silently, i.e. not add any auth header if the `auth_header` function does not exist



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2644621190


##########
pyiceberg/catalog/__init__.py:
##########
@@ -77,6 +77,7 @@
 _ENV_CONFIG = Config()
 
 TOKEN = "token"
+AUTH_MANAGER = "auth.manager"

Review Comment:
   nit, could we move this to `pyiceberg/catalog/rest/auth.py`? with the other auth manager code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "010Soham (via GitHub)" <gi...@apache.org>.
010Soham commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2663736963


##########
pyiceberg/io/fsspec.py:
##########
@@ -121,9 +121,19 @@ def __call__(self, request: "AWSRequest", **_: Any) -> None:
         signer_url = self.properties.get(S3_SIGNER_URI, self.properties[URI]).rstrip("/")  # type: ignore
         signer_endpoint = self.properties.get(S3_SIGNER_ENDPOINT, S3_SIGNER_ENDPOINT_DEFAULT)
 
-        signer_headers = {}
+        signer_headers: dict[str, str] = {}
+
+        auth_header: str | None = None
         if token := self.properties.get(TOKEN):
-            signer_headers = {"Authorization": f"Bearer {token}"}
+            auth_header = f"Bearer {token}"

Review Comment:
   that sounds good, lets address the cleaner design in #2862 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "010Soham (via GitHub)" <gi...@apache.org>.
010Soham commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2663920187


##########
pyiceberg/io/fsspec.py:
##########
@@ -121,9 +121,19 @@ def __call__(self, request: "AWSRequest", **_: Any) -> None:
         signer_url = self.properties.get(S3_SIGNER_URI, self.properties[URI]).rstrip("/")  # type: ignore
         signer_endpoint = self.properties.get(S3_SIGNER_ENDPOINT, S3_SIGNER_ENDPOINT_DEFAULT)
 
-        signer_headers = {}
+        signer_headers: dict[str, str] = {}
+
+        auth_header: str | None = None
         if token := self.properties.get(TOKEN):
-            signer_headers = {"Authorization": f"Bearer {token}"}
+            auth_header = f"Bearer {token}"
+        elif auth_manager := self.properties.get(AUTH_MANAGER):
+            header = getattr(auth_manager, "auth_header", None)
+            if callable(header):
+                auth_header = header()

Review Comment:
   updated to call `auth_manager.auth_header ( )` directly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2645920570


##########
pyiceberg/io/fsspec.py:
##########
@@ -121,9 +121,19 @@ def __call__(self, request: "AWSRequest", **_: Any) -> None:
         signer_url = self.properties.get(S3_SIGNER_URI, self.properties[URI]).rstrip("/")  # type: ignore
         signer_endpoint = self.properties.get(S3_SIGNER_ENDPOINT, S3_SIGNER_ENDPOINT_DEFAULT)
 
-        signer_headers = {}
+        signer_headers: dict[str, str] = {}
+
+        auth_header: str | None = None
         if token := self.properties.get(TOKEN):
-            signer_headers = {"Authorization": f"Bearer {token}"}
+            auth_header = f"Bearer {token}"

Review Comment:
   This may be out of scope for this bug-fix PR, but it looks like we’re tightly coupling `AuthManager` and authentication tokens which are `RestCatalog` concepts into `S3FileIO`, which should be `Catalog` type agnostic.
   
   It might be worth revisiting this design in more detail in the future to ensure we don’t introduce fallback logic that’s driven by configuration properties rather than clearer separation of concerns



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#discussion_r2646205973


##########
pyiceberg/io/fsspec.py:
##########
@@ -121,9 +121,19 @@ def __call__(self, request: "AWSRequest", **_: Any) -> None:
         signer_url = self.properties.get(S3_SIGNER_URI, self.properties[URI]).rstrip("/")  # type: ignore
         signer_endpoint = self.properties.get(S3_SIGNER_ENDPOINT, S3_SIGNER_ENDPOINT_DEFAULT)
 
-        signer_headers = {}
+        signer_headers: dict[str, str] = {}
+
+        auth_header: str | None = None
         if token := self.properties.get(TOKEN):
-            signer_headers = {"Authorization": f"Bearer {token}"}
+            auth_header = f"Bearer {token}"

Review Comment:
   in case we forget, https://github.com/apache/iceberg-python/issues/2862 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Pass REST auth manager to S3 signer [iceberg-python]

Posted by "010Soham (via GitHub)" <gi...@apache.org>.
010Soham commented on PR #2846:
URL: https://github.com/apache/iceberg-python/pull/2846#issuecomment-3773138645

   Glad it was helpful @kevinjqliu  😄


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org