You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "fcfangcc (via GitHub)" <gi...@apache.org> on 2025/09/25 03:04:41 UTC
[PR] Support fsspec s3 addressing_style properties [iceberg-python]
fcfangcc opened a new pull request, #2517:
URL: https://github.com/apache/iceberg-python/pull/2517
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
Impl #2516
Then changed it like
```python
catalog = load_catalog(
"docs",
**{
"type": "rest",
"uri": "http://lakekeeper:8181/catalog",
"s3.addressing-style": "virtual" ,
...
},
)
```
## Are these changes tested?
Add test case for fsspec s3 addressing_style properties
## Are there any user-facing changes?
The default behavior is not modified
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support fsspec s3 addressing_style properties [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #2517:
URL: https://github.com/apache/iceberg-python/pull/2517#discussion_r2380097088
##########
pyiceberg/io/__init__.py:
##########
Review Comment:
I see we already have this one, should we re-use that one? I'm not sure why it isn't being used 🤔 Seems like a faulty refactor somewhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support fsspec s3 addressing_style properties [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2517:
URL: https://github.com/apache/iceberg-python/pull/2517#discussion_r2380770010
##########
pyiceberg/io/fsspec.py:
##########
@@ -168,6 +169,9 @@ def _s3(properties: Properties) -> AbstractFileSystem:
if request_timeout := properties.get(S3_REQUEST_TIMEOUT):
config_kwargs["read_timeout"] = float(request_timeout)
+ if _force_virtual_addressing := properties.get(S3_FORCE_VIRTUAL_ADDRESSING):
+ config_kwargs["s3"] = {"addressing_style": "virtual"}
Review Comment:
👍
`config_kwargs` is passed to `botocore.client.Config`
https://s3fs.readthedocs.io/en/latest/api.html#s3fs.core.S3FileSystem
`botocore.client.Config` expects `botocore.client.Config` in `s3` dict
https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support fsspec s3 addressing_style properties [iceberg-python]
Posted by "fcfangcc (via GitHub)" <gi...@apache.org>.
fcfangcc commented on code in PR #2517:
URL: https://github.com/apache/iceberg-python/pull/2517#discussion_r2380500746
##########
pyiceberg/io/__init__.py:
##########
Review Comment:
> I see we already have this one, should we re-use that one? I'm not sure why it isn't being used 🤔 Seems like a faulty refactor somewhere
Because two different S3 dependency libraries are used, the configuration parameters for this in the two libraries are different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support fsspec s3 addressing_style properties [iceberg-python]
Posted by "fcfangcc (via GitHub)" <gi...@apache.org>.
fcfangcc commented on PR #2517:
URL: https://github.com/apache/iceberg-python/pull/2517#issuecomment-3336449376
@Fokko I modified and used the same parameter **s3.force-virtual-addressing** to changed addressing_style
In fsspec impl, s3 api dependent s3fs -> aiobotocore, this lib use **s3.addressing_style** to control.
But in pyarrow impl, s3 api dependent **from pyarrow.fs import S3FileSystem**, this lib use **force-virtual-addressing** to control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support fsspec s3 addressing_style properties [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2517:
URL: https://github.com/apache/iceberg-python/pull/2517#issuecomment-3336703928
Thanks for the fix @fcfangcc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support fsspec s3 addressing_style properties [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #2517:
URL: https://github.com/apache/iceberg-python/pull/2517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org