You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "stats-dev (via GitHub)" <gi...@apache.org> on 2026/01/25 08:56:50 UTC
[PR] AWS profile support [iceberg-python]
stats-dev opened a new pull request, #2948:
URL: https://github.com/apache/iceberg-python/pull/2948
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
Closes #2841
# Rationale for this change
This PR adds explicit AWS profile support for both the Glue catalog client and
fsspec-based S3 FileIO.
While `GlueCatalog` already supports profile configuration, fsspec-based S3
operations did not propagate profile selection to the underlying
`S3FileSystem` or async AWS session. As a result, users had to rely on environment
variables or the default AWS profile, which makes it difficult to work with
multiple AWS configurations in parallel.
This change introduces two configuration properties:
- `client.profile-name`: a unified AWS profile for the catalog client and FileIO
- `s3.profile-name`: an AWS profile specifically for S3 FileIO
Profile resolution follows this precedence:
1. `s3.profile-name`
2. `client.profile-name`
This ensures consistent and explicit credential selection across catalog and
FileIO layers when using the fsspec backend.
## Are these changes tested?
Yes. New unit tests were added to validate the profile propagation behavior.
- **Glue Catalog**
- Verifies that `boto3.Session(profile_name=...)` is created when initializing
`GlueCatalog` with `client.profile-name`.
- **S3 FileIO (fsspec)**
- Verifies that `client.profile-name` or `s3.profile-name` results in the
creation of an async AWS session with the correct profile, which is then
passed to `S3FileSystem`.
The tests were run locally with:
```bash
pytest tests/catalog/test_glue_profile.py tests/io/test_fsspec_profile.py
```
Output would be:
```
==================== test session starts =====================
platform darwin -- Python 3.12.4, pytest-9.0.2, pluggy-1.6.0
rootdir: ${ROOTDIR}/iceberg-python
configfile: pyproject.toml
plugins: anyio-4.2.0, lazy-fixture-0.6.3, requests-mock-1.12.1
collected 3 items
tests/catalog/test_glue_profile.py . [ 33%]
tests/io/test_fsspec_profile.py .. [100%]
===================== 3 passed in 1.02s ======================
```
## Are there any user-facing changes?
Yes, this adds new configuration properties that users can set:
- `client.profile-name`: Sets the AWS profile for both the catalog client and FileIO (unified configuration).
- `s3.profile-name`: Sets the AWS profile specifically for S3 FileIO.
**Example Usage:**
```python
catalog = GlueCatalog(
"my_catalog",
**{
"client.profile-name": "my-aws-profile",
# ... other config
}
)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2726301003
##########
tests/io/test_fsspec_profile.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import uuid
+from unittest import mock
+
+from pyiceberg.io.fsspec import FsspecFileIO
+from pyiceberg.typedef import Properties
+
+UNIFIED_AWS_SESSION_PROPERTIES = {
+ "client.access-key-id": "client.access-key-id",
+ "client.secret-access-key": "client.secret-access-key",
+ "client.region": "client.region",
+ "client.session-token": "client.session-token",
+}
+
+
+def test_fsspec_s3_session_properties_with_profile() -> None:
+ session_properties: Properties = {
+ "s3.profile-name": "test-profile",
+ "s3.endpoint": "http://localhost:9000",
+ **UNIFIED_AWS_SESSION_PROPERTIES,
+ }
+
+ with mock.patch("s3fs.S3FileSystem") as mock_s3fs, mock.patch("aiobotocore.session.AioSession") as mock_aio_session:
+ s3_fileio = FsspecFileIO(properties=session_properties)
+ filename = str(uuid.uuid4())
+
+ s3_fileio.new_input(location=f"s3://warehouse/{filename}")
+
+ mock_aio_session.assert_called_with(profile="test-profile")
+ mock_s3fs.assert_called_with(
+ anon=False,
+ client_kwargs={
+ "endpoint_url": "http://localhost:9000",
+ "aws_access_key_id": "client.access-key-id",
+ "aws_secret_access_key": "client.secret-access-key",
+ "region_name": "client.region",
+ "aws_session_token": "client.session-token",
+ },
+ config_kwargs={},
+ session=mock_aio_session(),
+ )
+
+
+def test_fsspec_s3_session_properties_with_client_profile() -> None:
+ session_properties: Properties = {
+ "client.profile-name": "test-profile",
+ "s3.endpoint": "http://localhost:9000",
+ **UNIFIED_AWS_SESSION_PROPERTIES,
+ }
+
+ with mock.patch("s3fs.S3FileSystem") as mock_s3fs, mock.patch("aiobotocore.session.AioSession") as mock_aio_session:
+ s3_fileio = FsspecFileIO(properties=session_properties)
+ filename = str(uuid.uuid4())
+
+ s3_fileio.new_input(location=f"s3://warehouse/{filename}")
+
+ mock_aio_session.assert_called_with(profile="test-profile")
+ mock_s3fs.assert_called_with(
+ anon=False,
+ client_kwargs={
+ "endpoint_url": "http://localhost:9000",
+ "aws_access_key_id": "client.access-key-id",
+ "aws_secret_access_key": "client.secret-access-key",
+ "region_name": "client.region",
+ "aws_session_token": "client.session-token",
+ },
+ config_kwargs={},
+ session=mock_aio_session(),
+ )
Review Comment:
I added `test_fsspec_s3_session_properties_with_s3_and_client_profile` function in `test_fsspec_profile.py`. It cloud verify that `s3.profile-name` takes precedence over `client.profile-name`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2726253171
##########
pyiceberg/io/__init__.py:
##########
@@ -41,12 +41,14 @@
logger = logging.getLogger(__name__)
+AWS_PROFILE_NAME = "client.profile-name"
AWS_REGION = "client.region"
AWS_ACCESS_KEY_ID = "client.access-key-id"
AWS_SECRET_ACCESS_KEY = "client.secret-access-key"
AWS_SESSION_TOKEN = "client.session-token"
AWS_ROLE_ARN = "client.role-arn"
AWS_ROLE_SESSION_NAME = "client.role-session-name"
+S3_PROFILE_NAME = "s3.profile-name"
Review Comment:
```
AWS_PROFILE_NAME = "client.profile-name"
S3_PROFILE_NAME = "s3.profile-name"
```
I think they are not the same parameter. `S3_PROFILE_NAME` ("s3.profile-name") and `AWS_PROFILE_NAME` ("client.profile-name") need to remain separate as they represent distinct configuration keys.
(S3-specific vs. global)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support to glue and fsspec s3 fileio [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2730096972
##########
pyiceberg/io/fsspec.py:
##########
@@ -205,7 +207,16 @@ def _s3(properties: Properties) -> AbstractFileSystem:
else:
anon = False
- fs = S3FileSystem(anon=anon, client_kwargs=client_kwargs, config_kwargs=config_kwargs)
+ s3_fs_kwargs = {
+ "anon": anon,
+ "client_kwargs": client_kwargs,
+ "config_kwargs": config_kwargs,
+ }
+
+ if profile_name := get_first_property_value(properties, S3_PROFILE_NAME, AWS_PROFILE_NAME):
+ s3_fs_kwargs["profile"] = profile_name
Review Comment:
Thanks for updating your feedback!
I performed local verification using a script without mocks against real my AWS profiles. I made two IAM Users with different permission profile(s3-only-allowed vs glue-only-allowed) and tested in local script.
First, I verified this locally using real AWS credentials.
- S3 (fsspec): using `s3.profile-name`=s3-only-allowed successfully wrote to a test S3 bucket. Switching to a different profile without S3 permissions resulted in `AccessDenied`, confirming the profile is actually used.
- Glue: Glue catalog calls (ex, list_namespaces) succeeded only when using the glue-only-allowed profile. Using a profile without Glue permissions resulted in an authorization failure.
So, using distinct profiles (ex, s3-allowed vs glue-allowed) correctly isolates permissions — the client succeeded only for the service allowed by the respective profile.
Second, I confirmed that providing a non-existent profile correctly raises botocore.exceptions.ProfileNotFound from both Glue and S3 clients.
Third, I checked that specific properties like s3.profile-name and glue.profile-name correctly override client.profile-name in a real environment..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2726255870
##########
pyiceberg/io/__init__.py:
##########
@@ -41,12 +41,14 @@
logger = logging.getLogger(__name__)
+AWS_PROFILE_NAME = "client.profile-name"
AWS_REGION = "client.region"
AWS_ACCESS_KEY_ID = "client.access-key-id"
AWS_SECRET_ACCESS_KEY = "client.secret-access-key"
AWS_SESSION_TOKEN = "client.session-token"
AWS_ROLE_ARN = "client.role-arn"
AWS_ROLE_SESSION_NAME = "client.role-session-name"
+S3_PROFILE_NAME = "s3.profile-name"
Review Comment:
I've already updated the S3 FileIO documentation to include s3.profile-name in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support to glue and fsspec s3 fileio [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2729237996
##########
pyiceberg/io/fsspec.py:
##########
@@ -205,7 +207,16 @@ def _s3(properties: Properties) -> AbstractFileSystem:
else:
anon = False
- fs = S3FileSystem(anon=anon, client_kwargs=client_kwargs, config_kwargs=config_kwargs)
+ s3_fs_kwargs = {
+ "anon": anon,
+ "client_kwargs": client_kwargs,
+ "config_kwargs": config_kwargs,
+ }
+
+ if profile_name := get_first_property_value(properties, S3_PROFILE_NAME, AWS_PROFILE_NAME):
+ s3_fs_kwargs["profile"] = profile_name
Review Comment:
Thanks! Heres the docs for "profile", as reference
https://s3fs.readthedocs.io/en/latest/#credentials
```
You can specify a profile using s3fs.S3FileSystem(profile='PROFILE'). Otherwise sf3s will use authentication via boto environment variables.
```
https://github.com/fsspec/s3fs/issues/324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2726184453
##########
pyiceberg/io/fsspec.py:
##########
@@ -205,7 +207,13 @@ def _s3(properties: Properties) -> AbstractFileSystem:
else:
anon = False
- fs = S3FileSystem(anon=anon, client_kwargs=client_kwargs, config_kwargs=config_kwargs)
+ session = None
+ if profile_name := get_first_property_value(properties, S3_PROFILE_NAME, AWS_PROFILE_NAME):
+ from aiobotocore.session import AioSession
+
+ session = AioSession(profile=profile_name)
Review Comment:
I agree with your opinion! How about this suggestion including your feedback?
```
# Define arguments for S3FileSystem initialization
s3_fs_kwargs = {
"anon": anon,
"client_kwargs": client_kwargs,
"config_kwargs": config_kwargs,
}
# 1. Retrieve the profile name from properties
if profile_name := get_first_property_value(properties, S3_PROFILE_NAME, AWS_PROFILE_NAME):
s3_fs_kwargs["profile"] = profile_name # 2. Add 'profile' to the arguments if it exists
fs = S3FileSystem(**s3_fs_kwargs) # 3. Initialize S3FileSystem unpacking the arguments
```
I've refactored the code to pass the profile as a keyword argument to S3FileSystem instead, allowing s3fs to handle the session creation internally as intended. I also updated the tests to verify that the profile argument is passed correctly.
The implementation is now much cleaner. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2725968933
##########
pyiceberg/catalog/glue.py:
##########
@@ -329,7 +329,7 @@ def __init__(self, name: str, client: Optional["GlueClient"] = None, **propertie
retry_mode_prop_value = get_first_property_value(properties, GLUE_RETRY_MODE)
session = boto3.Session(
- profile_name=properties.get(GLUE_PROFILE_NAME),
+ profile_name=properties.get(GLUE_PROFILE_NAME, properties.get(AWS_PROFILE_NAME)),
Review Comment:
Thanks!
I'll switch this to use `get_first_property_value` to keep the behavior consistent with the rest of the codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2726280408
##########
pyiceberg/io/__init__.py:
##########
@@ -41,12 +41,14 @@
logger = logging.getLogger(__name__)
+AWS_PROFILE_NAME = "client.profile-name"
AWS_REGION = "client.region"
AWS_ACCESS_KEY_ID = "client.access-key-id"
AWS_SECRET_ACCESS_KEY = "client.secret-access-key"
AWS_SESSION_TOKEN = "client.session-token"
AWS_ROLE_ARN = "client.role-arn"
AWS_ROLE_SESSION_NAME = "client.role-session-name"
+S3_PROFILE_NAME = "s3.profile-name"
Review Comment:
also updated `configuration.md` to include `client.profile-name` in the Unified AWS Credentials table as requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2726152355
##########
pyiceberg/catalog/glue.py:
##########
Review Comment:
I added s3.profile-name and client.profile-name to the configuration docs.
while updating the docs, I noticed that the existing descriptions used "static profile" in table. I felt that "AWS profile" is more intuitive and accurate, so I updated the descriptions for glue, dynamodb, s3, and client tables to maintain consistency. if you prefer keeping it as "static" (instead of AWS profile..)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2726301003
##########
tests/io/test_fsspec_profile.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import uuid
+from unittest import mock
+
+from pyiceberg.io.fsspec import FsspecFileIO
+from pyiceberg.typedef import Properties
+
+UNIFIED_AWS_SESSION_PROPERTIES = {
+ "client.access-key-id": "client.access-key-id",
+ "client.secret-access-key": "client.secret-access-key",
+ "client.region": "client.region",
+ "client.session-token": "client.session-token",
+}
+
+
+def test_fsspec_s3_session_properties_with_profile() -> None:
+ session_properties: Properties = {
+ "s3.profile-name": "test-profile",
+ "s3.endpoint": "http://localhost:9000",
+ **UNIFIED_AWS_SESSION_PROPERTIES,
+ }
+
+ with mock.patch("s3fs.S3FileSystem") as mock_s3fs, mock.patch("aiobotocore.session.AioSession") as mock_aio_session:
+ s3_fileio = FsspecFileIO(properties=session_properties)
+ filename = str(uuid.uuid4())
+
+ s3_fileio.new_input(location=f"s3://warehouse/{filename}")
+
+ mock_aio_session.assert_called_with(profile="test-profile")
+ mock_s3fs.assert_called_with(
+ anon=False,
+ client_kwargs={
+ "endpoint_url": "http://localhost:9000",
+ "aws_access_key_id": "client.access-key-id",
+ "aws_secret_access_key": "client.secret-access-key",
+ "region_name": "client.region",
+ "aws_session_token": "client.session-token",
+ },
+ config_kwargs={},
+ session=mock_aio_session(),
+ )
+
+
+def test_fsspec_s3_session_properties_with_client_profile() -> None:
+ session_properties: Properties = {
+ "client.profile-name": "test-profile",
+ "s3.endpoint": "http://localhost:9000",
+ **UNIFIED_AWS_SESSION_PROPERTIES,
+ }
+
+ with mock.patch("s3fs.S3FileSystem") as mock_s3fs, mock.patch("aiobotocore.session.AioSession") as mock_aio_session:
+ s3_fileio = FsspecFileIO(properties=session_properties)
+ filename = str(uuid.uuid4())
+
+ s3_fileio.new_input(location=f"s3://warehouse/{filename}")
+
+ mock_aio_session.assert_called_with(profile="test-profile")
+ mock_s3fs.assert_called_with(
+ anon=False,
+ client_kwargs={
+ "endpoint_url": "http://localhost:9000",
+ "aws_access_key_id": "client.access-key-id",
+ "aws_secret_access_key": "client.secret-access-key",
+ "region_name": "client.region",
+ "aws_session_token": "client.session-token",
+ },
+ config_kwargs={},
+ session=mock_aio_session(),
+ )
Review Comment:
I added `test_fsspec_s3_session_properties_with_s3_and_client_profile` function in `test_fsspec_profile.py`. It could verify that `s3.profile-name` takes precedence over `client.profile-name`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support to glue and fsspec s3 fileio [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#issuecomment-3806373579
Thanks for the PR @stats-dev and thanks for the review @geruh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support to glue and fsspec s3 fileio [iceberg-python]
Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2730151831
##########
pyiceberg/io/fsspec.py:
##########
@@ -205,7 +207,16 @@ def _s3(properties: Properties) -> AbstractFileSystem:
else:
anon = False
- fs = S3FileSystem(anon=anon, client_kwargs=client_kwargs, config_kwargs=config_kwargs)
+ s3_fs_kwargs = {
+ "anon": anon,
+ "client_kwargs": client_kwargs,
+ "config_kwargs": config_kwargs,
+ }
+
+ if profile_name := get_first_property_value(properties, S3_PROFILE_NAME, AWS_PROFILE_NAME):
+ s3_fs_kwargs["profile"] = profile_name
Review Comment:
Also, changes looks good!
The cred resolving chain looks as expected with `s3.profile-name` --> `client.profile-name` using `get_first_property_value`, and you're only passing the profile kwarg when it's actually set, which avoids the `profile=None` issue. The Glue pattern matches what's already there for `region_name` and `aws_access_key_id`, so that's consistent.
Tests cover both the fallback and resolving chain scenarios, which is what matters. And thanks for testing with distinct IAM profiles in your actual setup. That's alot more useful than unit tests alone for catching edge cases. One thing to note is a user can set profile and credentials but credentials should take precedence here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2726283061
##########
tests/catalog/test_glue_profile.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+
+from moto import mock_aws
+
+from pyiceberg.catalog.glue import GlueCatalog
+from pyiceberg.typedef import Properties
+
+UNIFIED_AWS_SESSION_PROPERTIES = {
+ "client.access-key-id": "client.access-key-id",
+ "client.secret-access-key": "client.secret-access-key",
+ "client.region": "client.region",
+ "client.session-token": "client.session-token",
+}
Review Comment:
ok
##########
tests/io/test_fsspec_profile.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import uuid
+from unittest import mock
+
+from pyiceberg.io.fsspec import FsspecFileIO
+from pyiceberg.typedef import Properties
+
+UNIFIED_AWS_SESSION_PROPERTIES = {
+ "client.access-key-id": "client.access-key-id",
+ "client.secret-access-key": "client.secret-access-key",
+ "client.region": "client.region",
+ "client.session-token": "client.session-token",
+}
Review Comment:
I refactored tests to import `UNIFIED_AWS_SESSION_PROPERTIES` from `tests.conftest.py` to avoid duplication.. it was duplicated..! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support to glue and fsspec s3 fileio [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2725699847
##########
tests/io/test_fsspec_profile.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import uuid
+from unittest import mock
+
+from pyiceberg.io.fsspec import FsspecFileIO
+from pyiceberg.typedef import Properties
+
+UNIFIED_AWS_SESSION_PROPERTIES = {
+ "client.access-key-id": "client.access-key-id",
+ "client.secret-access-key": "client.secret-access-key",
+ "client.region": "client.region",
+ "client.session-token": "client.session-token",
+}
Review Comment:
The constant `UNIFIED_AWS_SESSION_PROPERTIES` is being redefined locally instead of importing it from `tests.conftest`, which deviates from the established pattern in the codebase. Multiple test files (e.g., `tests/catalog/test_glue.py`, `tests/catalog/test_dynamodb.py`, `tests/io/test_pyarrow.py`) import this constant from `tests.conftest`. Consider importing it from `tests.conftest` to maintain consistency and avoid duplication.
##########
tests/catalog/test_glue_profile.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+
+from moto import mock_aws
+
+from pyiceberg.catalog.glue import GlueCatalog
+from pyiceberg.typedef import Properties
+
+UNIFIED_AWS_SESSION_PROPERTIES = {
+ "client.access-key-id": "client.access-key-id",
+ "client.secret-access-key": "client.secret-access-key",
+ "client.region": "client.region",
+ "client.session-token": "client.session-token",
+}
+
+
+@mock_aws
+def test_passing_client_profile_name_properties_to_glue() -> None:
+ session_properties: Properties = {
+ "client.profile-name": "profile_name",
+ **UNIFIED_AWS_SESSION_PROPERTIES,
+ }
+
+ with mock.patch("boto3.Session") as mock_session:
+ test_catalog = GlueCatalog("glue", **session_properties)
+
+ mock_session.assert_called_with(
+ aws_access_key_id="client.access-key-id",
+ aws_secret_access_key="client.secret-access-key",
+ aws_session_token="client.session-token",
+ region_name="client.region",
+ profile_name="profile_name",
+ botocore_session=None,
+ )
+ assert test_catalog.glue is mock_session().client()
Review Comment:
Missing test case for precedence when both `glue.profile-name` and `client.profile-name` are provided. According to the implementation and pattern established in the codebase, `glue.profile-name` should take precedence over `client.profile-name`. A test should verify that when both properties are set, `glue.profile-name` is used.
##########
tests/io/test_fsspec.py:
##########
@@ -34,7 +34,13 @@
from pyiceberg.io.fsspec import FsspecFileIO, S3V4RestSigner
from pyiceberg.io.pyarrow import PyArrowFileIO
from pyiceberg.typedef import Properties
-from tests.conftest import UNIFIED_AWS_SESSION_PROPERTIES
+
+UNIFIED_AWS_SESSION_PROPERTIES = {
+ "client.access-key-id": "client.access-key-id",
+ "client.secret-access-key": "client.secret-access-key",
+ "client.region": "client.region",
+ "client.session-token": "client.session-token",
+}
Review Comment:
The constant `UNIFIED_AWS_SESSION_PROPERTIES` is being redefined locally instead of importing it from `tests.conftest`, which deviates from the established pattern in the codebase. Other test files in the same directory (e.g., `tests/io/test_pyarrow.py`) import this constant from `tests.conftest`. Consider importing it from `tests.conftest` to maintain consistency and avoid duplication.
##########
pyiceberg/io/__init__.py:
##########
@@ -41,12 +41,14 @@
logger = logging.getLogger(__name__)
+AWS_PROFILE_NAME = "client.profile-name"
AWS_REGION = "client.region"
AWS_ACCESS_KEY_ID = "client.access-key-id"
AWS_SECRET_ACCESS_KEY = "client.secret-access-key"
AWS_SESSION_TOKEN = "client.session-token"
AWS_ROLE_ARN = "client.role-arn"
AWS_ROLE_SESSION_NAME = "client.role-session-name"
+S3_PROFILE_NAME = "s3.profile-name"
Review Comment:
The documentation for Unified AWS Credentials (mkdocs/docs/configuration.md lines 808-831) should be updated to include the newly added `client.profile-name` property. The table starting at line 823 is missing an entry for `client.profile-name` to document that it sets the AWS profile for both the catalog and S3 FileIO.
```suggestion
S3_PROFILE_NAME = AWS_PROFILE_NAME
```
##########
tests/io/test_fsspec_profile.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import uuid
+from unittest import mock
+
+from pyiceberg.io.fsspec import FsspecFileIO
+from pyiceberg.typedef import Properties
+
+UNIFIED_AWS_SESSION_PROPERTIES = {
+ "client.access-key-id": "client.access-key-id",
+ "client.secret-access-key": "client.secret-access-key",
+ "client.region": "client.region",
+ "client.session-token": "client.session-token",
+}
+
+
+def test_fsspec_s3_session_properties_with_profile() -> None:
+ session_properties: Properties = {
+ "s3.profile-name": "test-profile",
+ "s3.endpoint": "http://localhost:9000",
+ **UNIFIED_AWS_SESSION_PROPERTIES,
+ }
+
+ with mock.patch("s3fs.S3FileSystem") as mock_s3fs, mock.patch("aiobotocore.session.AioSession") as mock_aio_session:
+ s3_fileio = FsspecFileIO(properties=session_properties)
+ filename = str(uuid.uuid4())
+
+ s3_fileio.new_input(location=f"s3://warehouse/{filename}")
+
+ mock_aio_session.assert_called_with(profile="test-profile")
+ mock_s3fs.assert_called_with(
+ anon=False,
+ client_kwargs={
+ "endpoint_url": "http://localhost:9000",
+ "aws_access_key_id": "client.access-key-id",
+ "aws_secret_access_key": "client.secret-access-key",
+ "region_name": "client.region",
+ "aws_session_token": "client.session-token",
+ },
+ config_kwargs={},
+ session=mock_aio_session(),
+ )
+
+
+def test_fsspec_s3_session_properties_with_client_profile() -> None:
+ session_properties: Properties = {
+ "client.profile-name": "test-profile",
+ "s3.endpoint": "http://localhost:9000",
+ **UNIFIED_AWS_SESSION_PROPERTIES,
+ }
+
+ with mock.patch("s3fs.S3FileSystem") as mock_s3fs, mock.patch("aiobotocore.session.AioSession") as mock_aio_session:
+ s3_fileio = FsspecFileIO(properties=session_properties)
+ filename = str(uuid.uuid4())
+
+ s3_fileio.new_input(location=f"s3://warehouse/{filename}")
+
+ mock_aio_session.assert_called_with(profile="test-profile")
+ mock_s3fs.assert_called_with(
+ anon=False,
+ client_kwargs={
+ "endpoint_url": "http://localhost:9000",
+ "aws_access_key_id": "client.access-key-id",
+ "aws_secret_access_key": "client.secret-access-key",
+ "region_name": "client.region",
+ "aws_session_token": "client.session-token",
+ },
+ config_kwargs={},
+ session=mock_aio_session(),
+ )
Review Comment:
Missing test case for precedence when both `s3.profile-name` and `client.profile-name` are provided. According to the PR description, `s3.profile-name` should take precedence over `client.profile-name`. A test should verify that when both properties are set, `s3.profile-name` is used.
##########
tests/catalog/test_glue_profile.py:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+
+from moto import mock_aws
+
+from pyiceberg.catalog.glue import GlueCatalog
+from pyiceberg.typedef import Properties
+
+UNIFIED_AWS_SESSION_PROPERTIES = {
+ "client.access-key-id": "client.access-key-id",
+ "client.secret-access-key": "client.secret-access-key",
+ "client.region": "client.region",
+ "client.session-token": "client.session-token",
+}
Review Comment:
The constant `UNIFIED_AWS_SESSION_PROPERTIES` is being redefined locally instead of importing it from `tests.conftest`, which deviates from the established pattern in the codebase. Multiple test files (e.g., `tests/catalog/test_glue.py`, `tests/catalog/test_dynamodb.py`) import this constant from `tests.conftest`. Consider importing it from `tests.conftest` to maintain consistency and avoid duplication.
##########
pyiceberg/io/__init__.py:
##########
@@ -41,12 +41,14 @@
logger = logging.getLogger(__name__)
+AWS_PROFILE_NAME = "client.profile-name"
AWS_REGION = "client.region"
AWS_ACCESS_KEY_ID = "client.access-key-id"
AWS_SECRET_ACCESS_KEY = "client.secret-access-key"
AWS_SESSION_TOKEN = "client.session-token"
AWS_ROLE_ARN = "client.role-arn"
AWS_ROLE_SESSION_NAME = "client.role-session-name"
+S3_PROFILE_NAME = "s3.profile-name"
Review Comment:
The S3 FileIO configuration documentation (mkdocs/docs/configuration.md lines 112-131) should be updated to include the newly added `s3.profile-name` property. The table should include an entry for `s3.profile-name` to document that it sets the AWS profile specifically for S3 FileIO operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2726184453
##########
pyiceberg/io/fsspec.py:
##########
@@ -205,7 +207,13 @@ def _s3(properties: Properties) -> AbstractFileSystem:
else:
anon = False
- fs = S3FileSystem(anon=anon, client_kwargs=client_kwargs, config_kwargs=config_kwargs)
+ session = None
+ if profile_name := get_first_property_value(properties, S3_PROFILE_NAME, AWS_PROFILE_NAME):
+ from aiobotocore.session import AioSession
+
+ session = AioSession(profile=profile_name)
Review Comment:
I agree with your opinion.. following the original s3fs docs, then getting much simple code.
How about this suggestion including your feedback?
```
# Define arguments for S3FileSystem initialization
s3_fs_kwargs = {
"anon": anon,
"client_kwargs": client_kwargs,
"config_kwargs": config_kwargs,
}
# 1. Retrieve the profile name from properties
if profile_name := get_first_property_value(properties, S3_PROFILE_NAME, AWS_PROFILE_NAME):
s3_fs_kwargs["profile"] = profile_name # 2. Add 'profile' to the arguments if it exists
fs = S3FileSystem(**s3_fs_kwargs) # 3. Initialize S3FileSystem unpacking the arguments
```
I've refactored the code to pass the profile as a keyword argument to S3FileSystem instead, allowing s3fs to handle the session creation internally as intended. I also updated the tests to verify that the profile argument is passed correctly.
The implementation is now much cleaner. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2729237996
##########
pyiceberg/io/fsspec.py:
##########
@@ -205,7 +207,16 @@ def _s3(properties: Properties) -> AbstractFileSystem:
else:
anon = False
- fs = S3FileSystem(anon=anon, client_kwargs=client_kwargs, config_kwargs=config_kwargs)
+ s3_fs_kwargs = {
+ "anon": anon,
+ "client_kwargs": client_kwargs,
+ "config_kwargs": config_kwargs,
+ }
+
+ if profile_name := get_first_property_value(properties, S3_PROFILE_NAME, AWS_PROFILE_NAME):
+ s3_fs_kwargs["profile"] = profile_name
Review Comment:
docs for "profile"
https://s3fs.readthedocs.io/en/latest/#credentials
```
You can specify a profile using s3fs.S3FileSystem(profile='PROFILE'). Otherwise sf3s will use authentication via boto environment variables.
```
https://github.com/fsspec/s3fs/issues/324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support to glue and fsspec s3 fileio [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2730096972
##########
pyiceberg/io/fsspec.py:
##########
@@ -205,7 +207,16 @@ def _s3(properties: Properties) -> AbstractFileSystem:
else:
anon = False
- fs = S3FileSystem(anon=anon, client_kwargs=client_kwargs, config_kwargs=config_kwargs)
+ s3_fs_kwargs = {
+ "anon": anon,
+ "client_kwargs": client_kwargs,
+ "config_kwargs": config_kwargs,
+ }
+
+ if profile_name := get_first_property_value(properties, S3_PROFILE_NAME, AWS_PROFILE_NAME):
+ s3_fs_kwargs["profile"] = profile_name
Review Comment:
Thanks for updating your feedback!
I performed local verification using a script without mocks against real my AWS profiles. I made two IAM Users with different permission profile(s3-only-allowed vs glue-only-allowed) and tested.
First, I verified this locally using real AWS credentials.
- S3 (fsspec): using `s3.profile-name`=s3-only-allowed successfully wrote to a test S3 bucket. Switching to a different profile without S3 permissions resulted in `AccessDenied`, confirming the profile is actually used.
- Glue: Glue catalog calls (ex, list_namespaces) succeeded only when using the glue-only-allowed profile. Using a profile without Glue permissions resulted in an authorization failure.
So, using distinct profiles (ex, s3-allowed vs glue-allowed) correctly isolates permissions — the client succeeded only for the service allowed by the respective profile.
Second, I confirmed that providing a non-existent profile correctly raises botocore.exceptions.ProfileNotFound from both Glue and S3 clients.
Third, I checked that specific properties like s3.profile-name and glue.profile-name correctly override client.profile-name in a real environment..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2725691803
##########
pyiceberg/io/fsspec.py:
##########
@@ -205,7 +207,13 @@ def _s3(properties: Properties) -> AbstractFileSystem:
else:
anon = False
- fs = S3FileSystem(anon=anon, client_kwargs=client_kwargs, config_kwargs=config_kwargs)
+ session = None
+ if profile_name := get_first_property_value(properties, S3_PROFILE_NAME, AWS_PROFILE_NAME):
+ from aiobotocore.session import AioSession
+
+ session = AioSession(profile=profile_name)
Review Comment:
Passing in the `AioSession` here will override the internal session object
From the docs:
https://s3fs.readthedocs.io/en/latest/api.html#s3fs.core.S3FileSystem
"""
session (aiobotocore AioSession object to be used for all connections.) – **This session will be used inplace of creating a new session inside S3FileSystem**. For example: aiobotocore.session.AioSession(profile=’test_user’)
"""
I think we can pass in profile name as kwarg to `S3FileSystem`
The kwarg will be passed into the internal AioSession object
https://github.com/fsspec/s3fs/blob/56402cd2565c5fa2aa84020c716560b3db27e8cd/s3fs/core.py#L563-L565
WDYT?
##########
pyiceberg/catalog/glue.py:
##########
Review Comment:
could you add this new config to the docs
https://py.iceberg.apache.org/configuration/#s3
https://py.iceberg.apache.org/configuration/#glue-catalog
##########
pyiceberg/catalog/glue.py:
##########
@@ -329,7 +329,7 @@ def __init__(self, name: str, client: Optional["GlueClient"] = None, **propertie
retry_mode_prop_value = get_first_property_value(properties, GLUE_RETRY_MODE)
session = boto3.Session(
- profile_name=properties.get(GLUE_PROFILE_NAME),
+ profile_name=properties.get(GLUE_PROFILE_NAME, properties.get(AWS_PROFILE_NAME)),
Review Comment:
```suggestion
profile_name=get_first_property_value(properties, GLUE_PROFILE_NAME, AWS_PROFILE_NAME),
```
we have a helper function for this 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2726152355
##########
pyiceberg/catalog/glue.py:
##########
Review Comment:
I added s3.profile-name and client.profile-name to the configuration docs.
Plus, while updating the docs, I noticed that the existing descriptions used "static profile" in table. I felt that "AWS profile" is more intuitive and accurate, so I updated the descriptions for glue, dynamodb, s3, and client table.
sections to maintain consistency.
Let me know if you prefer keeping it as "static" (instead of AWS profile..). Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support to glue and fsspec s3 fileio [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2730096972
##########
pyiceberg/io/fsspec.py:
##########
@@ -205,7 +207,16 @@ def _s3(properties: Properties) -> AbstractFileSystem:
else:
anon = False
- fs = S3FileSystem(anon=anon, client_kwargs=client_kwargs, config_kwargs=config_kwargs)
+ s3_fs_kwargs = {
+ "anon": anon,
+ "client_kwargs": client_kwargs,
+ "config_kwargs": config_kwargs,
+ }
+
+ if profile_name := get_first_property_value(properties, S3_PROFILE_NAME, AWS_PROFILE_NAME):
+ s3_fs_kwargs["profile"] = profile_name
Review Comment:
Thanks for updating your feedback!
I performed local verification using a script without mocks against real my AWS profiles. I made two AWS IAM Users with different permission profile(s3-only-allowed vs glue-only-allowed) and tested.
First, I verified this locally using real AWS credentials.
- S3 (fsspec): using `s3.profile-name`=s3-only-allowed successfully wrote to a test S3 bucket. Switching to a different profile without S3 permissions resulted in `AccessDenied`, confirming the profile is actually used.
- Glue: Glue catalog calls (ex, list_namespaces) succeeded only when using the glue-only-allowed profile. Using a profile without Glue permissions resulted in an authorization failure.
So, using distinct profiles (ex, s3-allowed vs glue-allowed) correctly isolates permissions — the client succeeded only for the service allowed by the respective profile.
Second, I confirmed that providing a non-existent profile correctly raises botocore.exceptions.ProfileNotFound from both Glue and S3 clients.
Third, I checked that specific properties like `s3.profile-name` and `glue.profile-name` correctly override `client.profile-name` in a real environment..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS profile support to glue and fsspec s3 fileio [iceberg-python]
Posted by "stats-dev (via GitHub)" <gi...@apache.org>.
stats-dev commented on code in PR #2948:
URL: https://github.com/apache/iceberg-python/pull/2948#discussion_r2730096972
##########
pyiceberg/io/fsspec.py:
##########
@@ -205,7 +207,16 @@ def _s3(properties: Properties) -> AbstractFileSystem:
else:
anon = False
- fs = S3FileSystem(anon=anon, client_kwargs=client_kwargs, config_kwargs=config_kwargs)
+ s3_fs_kwargs = {
+ "anon": anon,
+ "client_kwargs": client_kwargs,
+ "config_kwargs": config_kwargs,
+ }
+
+ if profile_name := get_first_property_value(properties, S3_PROFILE_NAME, AWS_PROFILE_NAME):
+ s3_fs_kwargs["profile"] = profile_name
Review Comment:
Thanks for updating your feedback!
I performed local verification using a script without mocks against real my AWS profiles. I made two AWS IAM Users with different permission profile(s3-only-allowed vs glue-only-allowed) and tested.
First, I verified this locally using real AWS credentials.
- S3 (fsspec): using `s3.profile-name`=s3-only-allowed successfully wrote to a test S3 bucket. Switching to a different profile without S3 permissions resulted in `AccessDenied`, confirming the profile is actually used.
- Glue: Glue catalog calls (ex, list_namespaces) succeeded only when using the glue-only-allowed profile. Using a profile without Glue permissions resulted in an authorization failure.
So, using distinct profiles (ex, s3-allowed vs glue-allowed) correctly isolates permissions — the client succeeded only for the service allowed by the respective profile.
Second, I confirmed that providing a non-existent profile correctly raises botocore.exceptions.ProfileNotFound from both Glue and S3 clients.
Third, I checked that specific properties like s3.profile-name and glue.profile-name correctly override client.profile-name in a real environment..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org