You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "c-thiel (via GitHub)" <gi...@apache.org> on 2024/05/14 21:11:24 UTC
[PR] Update fsspec.py to respect s3.signer.uri property [iceberg-python]
c-thiel opened a new pull request, #741:
URL: https://github.com/apache/iceberg-python/pull/741
Fixes https://github.com/apache/iceberg-python/issues/740
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Update fsspec.py to respect s3.signer.uri property [iceberg-python]
Posted by "c-thiel (via GitHub)" <gi...@apache.org>.
c-thiel commented on PR #741:
URL: https://github.com/apache/iceberg-python/pull/741#issuecomment-2127322592
@Fokko I added docs and the constant.
The constant is a very good idea - I hope we will be able to use remote signing with FileIO as well eventually. Right now only the fsspec impl. respects it.
I saw that tabular.io actually provides explicit S3 credentials (on top of remote signing), presumably via AWS STS, if "vended-credentials" are requested (https://github.com/apache/iceberg/blob/b3c25fb7608934d975a054b353823ca001ca3742/open-api/rest-catalog-open-api.yaml#L1495). This is something that can only ever work for AWS S3 and is noticeably slower than using remote signing. As remote signing works also with on-prem deployments, I really hope this is going to become the default for all clients and not vended-credentials. tabular does this only for pyiceberg. Spark requests remote-signing so there is no need to go the extra mile and generate S3 creds.
Right now unfortunately in pyiceberg, "vended-credentials" is hardcoded
https://github.com/apache/iceberg-python/blob/42afc439d362ef1b3dcff03a1ffd959bc0a399ca/pyiceberg/catalog/rest.py#L501
, even though "remote-signing" is actually supported via fsspec. If the server decides to just ignore what the client requests and push remote signing anyway together with:
"rest.sigv4-enabled": "true",
"py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO",
it works like a charm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Update fsspec.py to respect s3.signer.uri property [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #741:
URL: https://github.com/apache/iceberg-python/pull/741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Update fsspec.py to respect s3.signer.uri property [iceberg-python]
Posted by "guitcastro (via GitHub)" <gi...@apache.org>.
guitcastro commented on PR #741:
URL: https://github.com/apache/iceberg-python/pull/741#issuecomment-2274893639
> @Fokko thanks a lot for your feedback - I added docs and the constant. The constant is a very good idea - I hope we will be able to use remote signing with FileIO as well eventually. Right now only the fsspec impl. respects it.
>
> I saw that tabular.io actually provides explicit S3 credentials (on top of remote signing), presumably via AWS STS, if "vended-credentials" are requested (https://github.com/apache/iceberg/blob/b3c25fb7608934d975a054b353823ca001ca3742/open-api/rest-catalog-open-api.yaml#L1495). This is something that can only ever work for AWS S3 and is noticeably slower than using remote signing. As remote signing works also with on-prem deployments, I really hope this is going to become the default for all clients and not vended-credentials. tabular does this only for pyiceberg. Spark requests remote-signing so there is no need to go the extra mile and generate S3 creds.
>
> Right now unfortunately in pyiceberg, "vended-credentials" is hardcoded
>
> https://github.com/apache/iceberg-python/blob/42afc439d362ef1b3dcff03a1ffd959bc0a399ca/pyiceberg/catalog/rest.py#L501
>
> , even though "remote-signing" is actually supported via fsspec. If the server decides to just ignore what the client requests and push remote signing anyway together with:
>
> ```
> "rest.sigv4-enabled": "true",
> "py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO",
> ```
>
> it works like a charm.
Unfortunately, for nessie, when using `X-Iceberg-Access-Delegation: vended-credentials` does not work. The endpoint does not return the `s3.signer.uri`. When the head is set to `remote-signing` it does return the correct value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Update fsspec.py to respect s3.signer.uri property [iceberg-python]
Posted by "c-thiel (via GitHub)" <gi...@apache.org>.
c-thiel commented on PR #741:
URL: https://github.com/apache/iceberg-python/pull/741#issuecomment-2119193916
@amogh-jahagirdar any thoughts about this PR? Would this be OK to incorporate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Update fsspec.py to respect s3.signer.uri property [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #741:
URL: https://github.com/apache/iceberg-python/pull/741#issuecomment-2127020569
@c-thiel Since it is in Java, I think it is fair to add it here. Two things:
- Can you mention it in the docs as well?
- Should we introduce a constant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Update fsspec.py to respect s3.signer.uri property [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #741:
URL: https://github.com/apache/iceberg-python/pull/741#issuecomment-2142322807
@c-thiel The idea is to support both. The problem is that with Arrow it is hard to inject in the signing process, since most of the code is pushed down to the C level, and there are no hooks to use a custom signer. Support for remote signing is something that we definitely want to continue to support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org