You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "snazy (via GitHub)" <gi...@apache.org> on 2026/01/28 14:00:34 UTC
[PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
snazy opened a new pull request, #3591:
URL: https://github.com/apache/polaris/pull/3591
This aligns the Ozone guide with the other object storages guides.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2757077736
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
So I built the image from your repo's main and it still work. Please see detail commands and output in https://gist.github.com/MonkeyCanCode/8c670106e1f3111f31a2982b512460e0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2737052471
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
Weird... Any idea on what changed in Polaris? I believe absent header should keep working... If behaviour changes it might be something we need to flag in docs :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2743287897
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
> static credentials passed to a client (Spark SQL shell in this case) [...]
My original example relied on local `AWS_*` env. variable, but I guess it was not very clear about that aspect, which prompted a fix by @MonkeyCanCode .
I'm personally fine with exposing example passwords in Spak session config since they are so obviously not meant for reuse and are exposed in the docker-compose files anyway.
That said, it should be possible to go back to `AWS_*` env. vars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2738539625
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
`Credential vending was requested for table [...], but no credentials are available` - this looks wrong in this context... It might be an unintended side effect of some other changes.
I believe if the delegation headers was not provided credential vending should be be assumed as "requested".
I'll keep investigation this.
I'd prefer a service code fix over a stop-gap empty here in the example, if we're ok with delaying this PR a bit more :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2737286239
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
I'll dig into this more... a bit later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2738118052
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
So I am not sure if this is truly needed as ozone doesn't support STS for credential vending and I had created https://github.com/apache/polaris/pull/3574 yesterday to add the credential. Without this header, it will just use the provided dummy credential.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2738234627
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
[See here](https://gist.github.com/snazy/20bd0e556a8a6724219bc9b3456bcab0#file-gistfile1-txt-L215) why the header's needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2737791483
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
If it helps you, @dimas-b, [this](https://gist.github.com/snazy/20bd0e556a8a6724219bc9b3456bcab0) is what happens without that header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2754237528
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
Even with an explicit Polaris server image build from "latest main", it is still failing with the same error message.
But it does _not_ fail with apache/polaris:latest (1.3.0) pulled from Docker Hub.
So there is definitely something odd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2737024415
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
Absent header should mean no credential vending :thinking:
Does it not work without the empty header?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2736992007
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
Why do we need an empty value here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2750361229
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
@snazy so I ran ur branch for the CI testing but with my local image (compiled from latest main from public repo) and it worked after removed the vended-credentials: https://gist.github.com/MonkeyCanCode/ff5dc40fca97cc122c503cd130b400c8
As you can see in the above:
```
...
::endgroup::
::group::Execution of Write SQL file for code block at line 53
::endgroup::
::group::Dump of Spark SQL shell code block at line 53
${SPARK_SQL_BIN} \
--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.0,org.apache.iceberg:iceberg-aws-bundle:1.9.0 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.polaris.type=rest \
--conf spark.sql.catalog.polaris.uri=http://localhost:8181/api/catalog \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
--conf spark.sql.catalog.polaris.credential=root:s3cr3t \
--conf spark.sql.catalog.polaris.client.region=us-west-2 \
--conf spark.sql.catalog.polaris.s3.access-key-id=polaris_root \
--conf spark.sql.catalog.polaris.s3.secret-access-key=polaris_pass \
-f "${BUILD_TESTS_DIR}/.current.sql"
rm -f "${BUILD_TESTS_DIR}/.current.sql"
...
Test summary:
✅ Test passed for it/../../site/content/guides/ozone/index.md (47.89s)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2737052471
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
Weird... Any idea on what changed in Polaris? I believe absent header should keep working... If behaviour changed it might be something we need to flag in docs :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2750355991
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
@snazy so I forked your branch and used the existed images from my node (compiled from latest main) and it worked: https://gist.github.com/MonkeyCanCode/ca0605de7968ab252c18ce338d216e46
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2742966004
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
I am not sure if the issue is related to timing as this is throws from Polaris as you said. Having credentials is for sure not ideal. However, for this case, as the backend doesn't support, I would think this is the right thing to do. For example, if the user is using ozone with securing S3, they would need to set these implicitly as vended credential won't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2738539625
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
`Credential vending was requested for table [...], but no credentials are available` - this looks wrong in this context... It might be an unintended side effect of some other changes.
I believe if the delegation headers was not provided credential vending should be be assumed as "requested".
I'll keep investigation this.
I'd prefer a service code fix over a stop-gap empty header in the example, if we're ok with delaying this PR a bit more :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2738909256
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
Instructions from https://github.com/apache/polaris/blob/cac26506c28176016775792bf02021569e0e5263/getting-started/ozone/README.md#L42 (with @MonkeyCanCode 's recent fix) work fine in my env. :shrug:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2737004217
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
So that Polaris does not vend any credentials.
See the ceph guide.
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
BTW: The Polaris behavior might have changed in the meantime, since the Ozone guide went initially in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2740704449
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
I'd prefer to have the guides-testing in, hence this PR.
Manual testing (copy paste from the Markdown) has different (timing) behavior than running the steps "as fast as possible" from the CI scripts.
The `Credential vending was requested for table [...], but no credentials are available` message is coming from Polaris, so I think this is an issue in the code base as it exists `main`.
Related/orthogonal: I'm not sure that using static credentials passed to a client (Spark SQL shell in this case) is a good option to demonstrate to users. What's missing here is S3 remote request signing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2755248085
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
Hmm, that is odd as I built from latest last week. Let me take another look if there are any other differences later tonight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2737043105
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2750355991
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
@snazy so I forked your branch and used the existed images from my node (compiled from latest main) and it worked: https://gist.github.com/MonkeyCanCode/ca0605de7968ab252c18ce338d216e46
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2743274448
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
I ran the Ozone getting-started example from `main` again and it worked fine without the `X-Iceberg-Access-Delegation` in Spark... :shrug:
Adding `X-Iceberg-Access-Delegation=""` to the README just does not feel right to me as it might be a source of confusion for users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2737283620
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
This looks like an innocent doc fix, but I believe we should really fix the core issue in this case :sweat_smile:
I do believe absent `X-Iceberg-Access-Delegation` should work as if no credential vending has been requested.
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
This looks like an innocent doc fix, but I believe we should really fix the core issue in this case :sweat_smile:
I do believe the absent `X-Iceberg-Access-Delegation` header should work as if no credential vending has been requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2749333035
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
> Adding `X-Iceberg-Access-Delegation=""` to the README just does not feel right to me as it might be a source of confusion for users.
@dimas-b it is [_still_ failing without that option](https://gist.github.com/snazy/4ccb9d85683029ef20f0f6a23b5bcb7b). If this option does not hurt others, I advocate to add it so the guide can be run by everybody.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2738524990
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
So it is working for me (which is how I validated my PR last night):
```
➜ spark-3.5.7-bin-hadoop3 bin/spark-sql \
--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.0,org.apache.iceberg:iceberg-aws-bundle:1.9.0 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.polaris.type=rest \
--conf spark.sql.catalog.polaris.uri=http://localhost:8181/api/catalog \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
--conf spark.sql.catalog.polaris.credential=root:s3cr3t \
--conf spark.sql.catalog.polaris.client.region=us-west-2 \
--conf spark.sql.catalog.polaris.s3.access-key-id=polaris_root \
--conf spark.sql.catalog.polaris.s3.secret-access-key=polaris_pass
:: loading settings :: url = jar:file:/Users/yong/Downloads/spark-3.5.7-bin-hadoop3/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /Users/yong/.ivy2/cache
The jars for the packages stored in: /Users/yong/.ivy2/jars
org.apache.iceberg#iceberg-spark-runtime-3.5_2.12 added as a dependency
org.apache.iceberg#iceberg-aws-bundle added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-89dd8f45-7ca0-4f4b-b3e5-aff97c570bb1;1.0
confs: [default]
found org.apache.iceberg#iceberg-spark-runtime-3.5_2.12;1.9.0 in central
found org.apache.iceberg#iceberg-aws-bundle;1.9.0 in central
:: resolution report :: resolve 76ms :: artifacts dl 3ms
:: modules in use:
org.apache.iceberg#iceberg-aws-bundle;1.9.0 from central in [default]
org.apache.iceberg#iceberg-spark-runtime-3.5_2.12;1.9.0 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 2 | 0 | 0 | 0 || 2 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-89dd8f45-7ca0-4f4b-b3e5-aff97c570bb1
confs: [default]
0 artifacts copied, 2 already retrieved (0kB/3ms)
26/01/28 14:12:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
26/01/28 14:12:11 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
26/01/28 14:12:11 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
26/01/28 14:12:11 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
26/01/28 14:12:11 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore yong@192.168.1.177
Spark Web UI available at http://192.168.1.177:4040
Spark master: local[*], Application Id: local-1769631130381
spark-sql (default)> use polaris;
26/01/28 14:12:13 WARN AuthManagers: Inferring rest.auth.type=oauth2 since property credential was provided. Please explicitly set rest.auth.type to avoid this warning.
26/01/28 14:12:13 WARN OAuth2Manager: Iceberg REST client is missing the OAuth2 server URI configuration and defaults to http://localhost:8181/api/catalog/v1/oauth/tokens. This automatic fallback will be removed in a future Iceberg release.It is recommended to configure the OAuth2 endpoint using the 'oauth2-server-uri' property to be prepared. This warning will disappear if the OAuth2 endpoint is explicitly configured. See https://github.com/apache/iceberg/issues/10537
26/01/28 14:12:14 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Time taken: 0.783 seconds
spark-sql ()> create namespace ns;
Time taken: 0.442 seconds
spark-sql ()> create table ns.t1 as select 'abc';
26/01/28 14:12:24 WARN GarbageCollectionMetrics: To enable non-built-in garbage collector(s) List(G1 Concurrent GC), users should configure it(them) to spark.eventLog.gcMetrics.youngGenerationGarbageCollectors or spark.eventLog.gcMetrics.oldGenerationGarbageCollectors
Time taken: 4.285 seconds
spark-sql ()> select * from ns.t1;
abc
Time taken: 0.579 seconds, Fetched 1 row(s)
spark-sql ()>
```
Is it possible the polaris image can be different on your local setup as we are just pointing to latest? Based on my understanding, this is why we have `stsUnavailable` set to true in this docker compose. I also tried the build the local latest 1.4.0-incubating and that still work as well. I am suspected some local changes to your latest polaris image. Do you mind double check by either remove the local image or rebuild with main branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2758400210
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
So this is weird: I accidentally purged literally all container images (didn't read the warning emitted by `docker image prune -a` :facepalm:) and after that it worked for me as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2737509113
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
What should it vend without STS?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2738539625
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
`Credential vending was requested for table [...], but no credentials are available` - this looks wrong in this context... It might be an unintended side effect of some other changes.
I believe if the delegation header was not provided, credential vending should be be assumed as "requested".
I'll keep investigation this.
I'd prefer a service code fix over a stop-gap empty header in the example, if we're ok with delaying this PR a bit more :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2737272879
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
Fact is, it doesn't work without the header.
I do not know why exactly now and the reason for it.
But one could figure out once we have #3553 ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy closed pull request #3591: Ozone guide: add missing `X-Iceberg-Access-Delegation` header
URL: https://github.com/apache/polaris/pull/3591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ozone guide: add missing `X-Iceberg-Access-Delegation` header [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #3591:
URL: https://github.com/apache/polaris/pull/3591#discussion_r2749550769
##########
getting-started/ozone/README.md:
##########
@@ -48,6 +48,7 @@ bin/spark-sql \
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
--conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
+ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation="" \
Review Comment:
This is very intriguing. I cannot reproduce the error locally. The error is generated here:
https://github.com/apache/polaris/blob/baf01a54696b28095a197cbde156d386913cfd3e/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java#L871-L887
As we see, this code path is clearly guarded by `if (delegationModes.contains(VENDED_CREDENTIALS))`.
My best guess is that _something_ is setting this header value to `vended-credentials` when running the Ozone getting-started guide, thus forcing some users to override it with an empty string.
I noticed that PyIceberg does set this header by default to `vended-credentials`:
https://github.com/apache/iceberg-python/blob/d62b36024ab36fc3f86be50634cd8b32ce3fb9a1/pyiceberg/catalog/rest/__init__.py#L768
But that still doesn't explain why we need to set it explicitly when running Spark SQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org