You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "MonkeyCanCode (via GitHub)" <gi...@apache.org> on 2024/10/23 03:57:25 UTC
[PR] Add doc for not supported operations for credential vending [polaris]
MonkeyCanCode opened a new pull request, #398:
URL: https://github.com/apache/polaris/pull/398
# Description
This is for adding doc around which operations are not supported by credential vending.
Fixes # (390)
## Type of change
Please delete options that are not relevant.
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] Documentation update
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [x] This change requires a documentation update
# Checklist:
Please delete options that are not relevant.
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add doc for not supported operations for credential vending [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #398:
URL: https://github.com/apache/polaris/pull/398#discussion_r1815366821
##########
site/content/in-dev/unreleased/overview.md:
##########
@@ -180,6 +180,12 @@ To secure interactions with service connections, Polaris vends temporary storage
execution. These credentials allow the query engine to run the query without requiring access to your external cloud storage for
Iceberg tables. This process is called credential vending.
+As of now, the following limitations are known regarding Apache Iceberg support:
+
+- **remove_orphan_files:** This operation does not currently support
+ credential vending when utilizing Apache Spark as the compute engine.
+ - For more details, please refer to [apache/iceberg#7914](https://github.com/apache/iceberg/pull/7914).
Review Comment:
@MonkeyCanCode thanks a lot for validating maintenance procedures. It's a good callout, but it is essentially an Spark issue. Maybe we can reword to something like this:
```
- **remove_orphan_files:** Apache Spark can't use credential vending for this due to a known issue. See [apache/iceberg#7914](https://github.com/apache/iceberg/pull/7914) for details.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add doc for not supported operations for credential vending [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain merged PR #398:
URL: https://github.com/apache/polaris/pull/398
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add doc for not supported operations for credential vending [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #398:
URL: https://github.com/apache/polaris/pull/398#discussion_r1813205664
##########
site/content/in-dev/unreleased/overview.md:
##########
@@ -180,6 +180,12 @@ To secure interactions with service connections, Polaris vends temporary storage
execution. These credentials allow the query engine to run the query without requiring access to your external cloud storage for
Iceberg tables. This process is called credential vending.
+As of now, the following limitations are known regarding Apache Iceberg support:
+
+- **remove_orphan_files:** This operation does not currently support
+ credential vending when utilizing Apache Spark as the compute engine.
+ - For more details, please refer to [apache/iceberg#7914](https://github.com/apache/iceberg/pull/7914).
Review Comment:
Thanks for the info. Will finish validations later today then update the PR with suggestions above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add doc for not supported operations for credential vending [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #398:
URL: https://github.com/apache/polaris/pull/398#discussion_r1815595277
##########
site/content/in-dev/unreleased/overview.md:
##########
@@ -180,6 +180,12 @@ To secure interactions with service connections, Polaris vends temporary storage
execution. These credentials allow the query engine to run the query without requiring access to your external cloud storage for
Iceberg tables. This process is called credential vending.
+As of now, the following limitations are known regarding Apache Iceberg support:
+
+- **remove_orphan_files:** This operation does not currently support
+ credential vending when utilizing Apache Spark as the compute engine.
+ - For more details, please refer to [apache/iceberg#7914](https://github.com/apache/iceberg/pull/7914).
Review Comment:
Anytime. Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add doc for not supported operations for credential vending [polaris]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #398:
URL: https://github.com/apache/polaris/pull/398#discussion_r1811896741
##########
site/content/in-dev/unreleased/overview.md:
##########
@@ -180,6 +180,12 @@ To secure interactions with service connections, Polaris vends temporary storage
execution. These credentials allow the query engine to run the query without requiring access to your external cloud storage for
Iceberg tables. This process is called credential vending.
+As of now, the following limitations are known regarding Apache Iceberg support:
+
+- **remove_orphan_files:** This operation does not currently support
+ credential vending when utilizing Apache Spark as the compute engine.
+ - For more details, please refer to [apache/iceberg#7914](https://github.com/apache/iceberg/pull/7914).
Review Comment:
@eric-maynard is right, I think it's better to use Hugo style link to have a clean documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add doc for not supported operations for credential vending [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #398:
URL: https://github.com/apache/polaris/pull/398#discussion_r1814165181
##########
site/content/in-dev/unreleased/overview.md:
##########
@@ -180,6 +180,12 @@ To secure interactions with service connections, Polaris vends temporary storage
execution. These credentials allow the query engine to run the query without requiring access to your external cloud storage for
Iceberg tables. This process is called credential vending.
+As of now, the following limitations are known regarding Apache Iceberg support:
+
+- **remove_orphan_files:** This operation does not currently support
+ credential vending when utilizing Apache Spark as the compute engine.
+ - For more details, please refer to [apache/iceberg#7914](https://github.com/apache/iceberg/pull/7914).
Review Comment:
From some quick read, Hugo style link is mainly for internal links within the site and external links seems to be preferred as markdown's style instead. Also, there are many places we are using markdown style links for external reference across current site. If that is more preferred, should we do it in a different PR instead?
Then in terms of table maintenance operations validation with credential vending, following are tested:
Trino:
- optimize: passed
- expire_snapshots: passed
- remove_orphan_files: passed
- drop_extended_stats: passed
Spark:
- expire_snapshots: passed
- remove_orphan_files: failed (not supported as mentioned in this PR)
- rewrite_data_files: passed
- rewrite_manifests: passed
- rewrite_position_delete_files: passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add doc for not supported operations for credential vending [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #398:
URL: https://github.com/apache/polaris/pull/398#discussion_r1811886356
##########
site/content/in-dev/unreleased/overview.md:
##########
@@ -180,6 +180,12 @@ To secure interactions with service connections, Polaris vends temporary storage
execution. These credentials allow the query engine to run the query without requiring access to your external cloud storage for
Iceberg tables. This process is called credential vending.
+As of now, the following limitations are known regarding Apache Iceberg support:
+
+- **remove_orphan_files:** This operation does not currently support
+ credential vending when utilizing Apache Spark as the compute engine.
+ - For more details, please refer to [apache/iceberg#7914](https://github.com/apache/iceberg/pull/7914).
Review Comment:
Should we be using the new Hugo-style links? I am honestly not clear on that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org