You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "helmiazizm (via GitHub)" <gi...@apache.org> on 2025/04/16 02:56:22 UTC
[PR] Fixed force_virtual_addressing problem [iceberg-python]
helmiazizm opened a new pull request, #1923:
URL: https://github.com/apache/iceberg-python/pull/1923
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
This fix changed the behavior for both `_oss_fs` and `_s3_fs` to be able to parse `s3.force-virtual-addressing` correctly.
# Are these changes tested?
No unit tests are given to this change yet
# Are there any user-facing changes?
No
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fixed force_virtual_addressing problem [iceberg-python]
Posted by "helmiazizm (via GitHub)" <gi...@apache.org>.
helmiazizm commented on code in PR #1923:
URL: https://github.com/apache/iceberg-python/pull/1923#discussion_r2049361189
##########
pyiceberg/io/pyarrow.py:
##########
@@ -408,7 +408,7 @@ def _initialize_oss_fs(self) -> FileSystem:
"access_key": get_first_property_value(self.properties, S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
"secret_key": get_first_property_value(self.properties, S3_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
"session_token": get_first_property_value(self.properties, S3_SESSION_TOKEN, AWS_SESSION_TOKEN),
- "region": get_first_property_value(self.properties, S3_REGION, AWS_REGION),
+ "force_virtual_addressing": property_as_bool(self.properties, S3_FORCE_VIRTUAL_ADDRESSING, True),
Review Comment:
Unlike S3 authentication, virtual addressing is mandatory in OSS. Not having it set to `True` will always throw access denied error, hence why setting it to `True` by default would make authentication easier for users in case if they ever forget to set `s3.force-virtual-addressing` parameter to `True`. I also don't think this setting for OSS will go away anytime soon as other S3-compatible object storages are also using the same method, so this change should be safe.
I still keep the `property_as_bool` function to make the authentication feel as graceful in case if users insist on setting it to `False`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fixed force_virtual_addressing problem [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1923:
URL: https://github.com/apache/iceberg-python/pull/1923#discussion_r2047668282
##########
pyiceberg/io/pyarrow.py:
##########
@@ -408,7 +408,7 @@ def _initialize_oss_fs(self) -> FileSystem:
"access_key": get_first_property_value(self.properties, S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
"secret_key": get_first_property_value(self.properties, S3_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
"session_token": get_first_property_value(self.properties, S3_SESSION_TOKEN, AWS_SESSION_TOKEN),
- "region": get_first_property_value(self.properties, S3_REGION, AWS_REGION),
+ "force_virtual_addressing": property_as_bool(self.properties, S3_FORCE_VIRTUAL_ADDRESSING, True),
Review Comment:
I don't think we can remove `region`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fixed force_virtual_addressing problem [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1923:
URL: https://github.com/apache/iceberg-python/pull/1923#discussion_r2047689979
##########
pyiceberg/io/pyarrow.py:
##########
@@ -408,7 +408,7 @@ def _initialize_oss_fs(self) -> FileSystem:
"access_key": get_first_property_value(self.properties, S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
"secret_key": get_first_property_value(self.properties, S3_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
"session_token": get_first_property_value(self.properties, S3_SESSION_TOKEN, AWS_SESSION_TOKEN),
- "region": get_first_property_value(self.properties, S3_REGION, AWS_REGION),
+ "force_virtual_addressing": property_as_bool(self.properties, S3_FORCE_VIRTUAL_ADDRESSING, True),
Review Comment:
Looks like it breaks the tests:
```
# For oss scheme, user provided region is used instead
> assert pyarrow_file_io.new_input(f"oss://{bucket_region[0]}/path/to/file")._filesystem.region == user_provided_region
E AssertionError: assert 'us-east-1' == 'ap-southeast-1'
E - ap-southeast-1
E + us-east-1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fixed force_virtual_addressing problem [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1923:
URL: https://github.com/apache/iceberg-python/pull/1923#discussion_r2049108649
##########
pyiceberg/io/pyarrow.py:
##########
@@ -408,7 +408,7 @@ def _initialize_oss_fs(self) -> FileSystem:
"access_key": get_first_property_value(self.properties, S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
"secret_key": get_first_property_value(self.properties, S3_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
"session_token": get_first_property_value(self.properties, S3_SESSION_TOKEN, AWS_SESSION_TOKEN),
- "region": get_first_property_value(self.properties, S3_REGION, AWS_REGION),
+ "force_virtual_addressing": property_as_bool(self.properties, S3_FORCE_VIRTUAL_ADDRESSING, True),
Review Comment:
@helmiazizm Thoughts on changing the default?
Also, below we have `False` the default:
```python
if self.properties.get(S3_FORCE_VIRTUAL_ADDRESSING) is not None:
client_kwargs["force_virtual_addressing"] = property_as_bool(self.properties, S3_FORCE_VIRTUAL_ADDRESSING, False)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fixed force_virtual_addressing problem [iceberg-python]
Posted by "helmiazizm (via GitHub)" <gi...@apache.org>.
helmiazizm commented on code in PR #1923:
URL: https://github.com/apache/iceberg-python/pull/1923#discussion_r2049361189
##########
pyiceberg/io/pyarrow.py:
##########
@@ -408,7 +408,7 @@ def _initialize_oss_fs(self) -> FileSystem:
"access_key": get_first_property_value(self.properties, S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
"secret_key": get_first_property_value(self.properties, S3_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
"session_token": get_first_property_value(self.properties, S3_SESSION_TOKEN, AWS_SESSION_TOKEN),
- "region": get_first_property_value(self.properties, S3_REGION, AWS_REGION),
+ "force_virtual_addressing": property_as_bool(self.properties, S3_FORCE_VIRTUAL_ADDRESSING, True),
Review Comment:
Unlike S3 authentication, virtual addressing is mandatory in OSS. Not having it set to `True` will always throw access denied error, hence why setting it to `True` by default would make authentication easier for users in case if they ever forget to set `s3.force-virtual-addressing` parameter to `True`. I also don't think this setting for OSS will go away anytime soon as other S3-compatible object storages are also using it.
I still keep the `property_as_bool` function to make the authentication feel as graceful in case if users insist on setting it to `False`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fixed force_virtual_addressing problem [iceberg-python]
Posted by "helmiazizm (via GitHub)" <gi...@apache.org>.
helmiazizm commented on code in PR #1923:
URL: https://github.com/apache/iceberg-python/pull/1923#discussion_r2048090964
##########
pyiceberg/io/pyarrow.py:
##########
@@ -408,7 +408,7 @@ def _initialize_oss_fs(self) -> FileSystem:
"access_key": get_first_property_value(self.properties, S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
"secret_key": get_first_property_value(self.properties, S3_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
"session_token": get_first_property_value(self.properties, S3_SESSION_TOKEN, AWS_SESSION_TOKEN),
- "region": get_first_property_value(self.properties, S3_REGION, AWS_REGION),
+ "force_virtual_addressing": property_as_bool(self.properties, S3_FORCE_VIRTUAL_ADDRESSING, True),
Review Comment:
> I don't think we can remove `region`?
For OSS authentication, it's not really necessary since they only need endpoint, but perhaps you're right that it's nicer to keep it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fixed force_virtual_addressing problem [iceberg-python]
Posted by "helmiazizm (via GitHub)" <gi...@apache.org>.
helmiazizm commented on code in PR #1923:
URL: https://github.com/apache/iceberg-python/pull/1923#discussion_r2049361189
##########
pyiceberg/io/pyarrow.py:
##########
@@ -408,7 +408,7 @@ def _initialize_oss_fs(self) -> FileSystem:
"access_key": get_first_property_value(self.properties, S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
"secret_key": get_first_property_value(self.properties, S3_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
"session_token": get_first_property_value(self.properties, S3_SESSION_TOKEN, AWS_SESSION_TOKEN),
- "region": get_first_property_value(self.properties, S3_REGION, AWS_REGION),
+ "force_virtual_addressing": property_as_bool(self.properties, S3_FORCE_VIRTUAL_ADDRESSING, True),
Review Comment:
Unlike S3 authentication, virtual addressing is mandatory in OSS. Not having it set to `True` will always throw access denied error, hence why setting it to `True` by default would make authentication easier for users in case if they ever forget to set `s3.force-virtual-addressing` parameter to `True`. I also don't think this setting for OSS will go away anytime soon as other S3-compatible object storages are also using it, so this change should be safe.
I still keep the `property_as_bool` function to make the authentication feel as graceful in case if users insist on setting it to `False`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fixed force_virtual_addressing problem [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1923:
URL: https://github.com/apache/iceberg-python/pull/1923#discussion_r2049488052
##########
pyiceberg/io/pyarrow.py:
##########
@@ -408,7 +408,7 @@ def _initialize_oss_fs(self) -> FileSystem:
"access_key": get_first_property_value(self.properties, S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
"secret_key": get_first_property_value(self.properties, S3_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
"session_token": get_first_property_value(self.properties, S3_SESSION_TOKEN, AWS_SESSION_TOKEN),
- "region": get_first_property_value(self.properties, S3_REGION, AWS_REGION),
+ "force_virtual_addressing": property_as_bool(self.properties, S3_FORCE_VIRTUAL_ADDRESSING, True),
Review Comment:
@helmiazizm Thanks for the explanation, I don't have any hands-on experience with Alibaba Cloud. Lets move this forward 👍
##########
pyiceberg/io/pyarrow.py:
##########
@@ -408,7 +408,7 @@ def _initialize_oss_fs(self) -> FileSystem:
"access_key": get_first_property_value(self.properties, S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
"secret_key": get_first_property_value(self.properties, S3_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
"session_token": get_first_property_value(self.properties, S3_SESSION_TOKEN, AWS_SESSION_TOKEN),
- "region": get_first_property_value(self.properties, S3_REGION, AWS_REGION),
+ "force_virtual_addressing": property_as_bool(self.properties, S3_FORCE_VIRTUAL_ADDRESSING, True),
Review Comment:
@helmiazizm Thanks for the explanation, as I don't have any hands-on experience with Alibaba Cloud. Lets move this forward 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fixed force_virtual_addressing problem [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #1923:
URL: https://github.com/apache/iceberg-python/pull/1923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fixed force_virtual_addressing problem [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1923:
URL: https://github.com/apache/iceberg-python/pull/1923#discussion_r2049108649
##########
pyiceberg/io/pyarrow.py:
##########
@@ -408,7 +408,7 @@ def _initialize_oss_fs(self) -> FileSystem:
"access_key": get_first_property_value(self.properties, S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
"secret_key": get_first_property_value(self.properties, S3_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
"session_token": get_first_property_value(self.properties, S3_SESSION_TOKEN, AWS_SESSION_TOKEN),
- "region": get_first_property_value(self.properties, S3_REGION, AWS_REGION),
+ "force_virtual_addressing": property_as_bool(self.properties, S3_FORCE_VIRTUAL_ADDRESSING, True),
Review Comment:
@helmiazizm Thoughts on changing the default?
Also, below we change the default:
```python
if self.properties.get(S3_FORCE_VIRTUAL_ADDRESSING) is not None:
client_kwargs["force_virtual_addressing"] = property_as_bool(self.properties, S3_FORCE_VIRTUAL_ADDRESSING, False)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org