You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "flyrain (via GitHub)" <gi...@apache.org> on 2025/04/06 19:06:08 UTC
[I] Document how Polaris work with different cloud storage(S3, Azure Blob Storage, GCS). [polaris]
flyrain opened a new issue, #1325:
URL: https://github.com/apache/polaris/issues/1325
### Is your feature request related to a problem? Please describe.
The topic was briefly mentioned in https://polaris.apache.org/in-dev/unreleased/configuration/ and https://polaris.apache.org/in-dev/unreleased/overview/#storage-configuration. They don't provide enough clear instruction how Polaris service provider to set them. Having a dedicated page with more detailed information will be helpful.
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document how Polaris works with different cloud storage(S3, Azure Blob Storage, GCS). [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on issue #1325:
URL: https://github.com/apache/polaris/issues/1325#issuecomment-4455881432
option 1 sounds good to me. Even the GCS page exists, https://polaris.apache.org/releases/1.4.1/configuration/configuring-polaris-for-production/configuring-gcs-cloud-storage-specific/, it needs a bit more work. The current content doesn't quite meet the stand that a system admin can follow it to config a Polaris server based on GCS from end to end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document how Polaris works with different cloud storage(S3, Azure Blob Storage, GCS). [polaris]
Posted by "mj006648 (via GitHub)" <gi...@apache.org>.
mj006648 commented on issue #1325:
URL: https://github.com/apache/polaris/issues/1325#issuecomment-4448579574
Quick update after looking at the existing docs structure.
What's already there:
- `getting-started/deploying-polaris/cloud-deploy/quickstart-deploy-{aws,azure,gcp}.md` — quick-start path (added in #1435)
- `configuration/configuring-polaris-for-production/configuring-gcs-cloud-storage-specific.md` — short server-side page for GCS
What seems missing for the end-to-end sysadmin perspective:
- AWS S3 and Azure Blob equivalents of the GCS production config page
- Client-side wiring (Spark / Trino / PyIceberg) for each backend
- A clear single entry point that walks an admin from server config to client connection to verification
Before I start, could you confirm which scope fits your intent:
Option 1 — minimal gap fill:
Add `configuring-aws-s3-cloud-storage-specific.md` and `configuring-azure-blob-cloud-storage-specific.md` mirroring the existing GCS page, server-side only. Client-side wiring and skill.md as separate follow-ups.
Option 2 — full E2E guide:
Per-backend pages covering server config, client connection, and verification in one place. Existing GCS page gets expanded to match. skill.md follows.
Option 1 lands faster; Option 2 is closer to the start-to-finish sysadmin walkthrough you described. Which one do you prefer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document how Polaris works with different cloud storage(S3, Azure Blob Storage, GCS). [polaris]
Posted by "AzazelSensei (via GitHub)" <gi...@apache.org>.
AzazelSensei commented on issue #1325:
URL: https://github.com/apache/polaris/issues/1325#issuecomment-5501271404
Opened https://github.com/apache/polaris/pull/5432 with the GCS production page expanded to the same sysadmin walkthrough as the S3 / Azure pages from #4451.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document how Polaris works with different cloud storage(S3, Azure Blob Storage, GCS). [polaris]
Posted by "Abhijoyshirovmukherjee (via GitHub)" <gi...@apache.org>.
Abhijoyshirovmukherjee commented on issue #1325:
URL: https://github.com/apache/polaris/issues/1325#issuecomment-2815460569
This is something necessary. Can anybody share any document or piece of how the polaris catalog would be integrated with the underlying storage to be queried with spark or flink.
Currently getting the following error while inserting or reading data from the iceberg table in blob using flink:
No File system for scheme: abfss
Note: Have tried to run it with spark with the same dependencies and it has worked for inserting but not for reading.
@flyrain
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document how Polaris works with different cloud storage(S3, Azure Blob Storage, GCS). [polaris]
Posted by "adnanhemani (via GitHub)" <gi...@apache.org>.
adnanhemani commented on issue #1325:
URL: https://github.com/apache/polaris/issues/1325#issuecomment-2852384660
Given #1435 merged, should we close this @flyrain ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document how Polaris works with different cloud storage(S3, Azure Blob Storage, GCS). [polaris]
Posted by "adnanhemani (via GitHub)" <gi...@apache.org>.
adnanhemani commented on issue #1325:
URL: https://github.com/apache/polaris/issues/1325#issuecomment-2828766495
@Abhijoyshirovmukherjee - can you confirm that this error is coming from Polaris and not the query engine itself? I've just put up a PR for this (#1435). Please feel free to take a look at how I've gotten this working for Spark and it may help you spot if there's anything that's missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document how Polaris works with different cloud storage(S3, Azure Blob Storage, GCS). [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on issue #1325:
URL: https://github.com/apache/polaris/issues/1325#issuecomment-4448043170
Thanks for picking this up, @mj006648. My idea is to provide end to end instructions that include both Polaris server side and client side configuration for different object storage systems, including AWS S3, Azure, and GCS. What you described here is already pretty close.
I’m thinking from the perspective of a system admin who wants to set up Polaris with a specific storage backend. The docs should clearly describe the recommended setup process from start to finish.
Beside a doc, another option could be to provide a skill.md so an agent can help automate the setup process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document how Polaris works with different cloud storage(S3, Azure Blob Storage, GCS). [polaris]
Posted by "frHimanshu (via GitHub)" <gi...@apache.org>.
frHimanshu commented on issue #1325:
URL: https://github.com/apache/polaris/issues/1325#issuecomment-2827389838
I'm up to give a pr on this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document how Polaris works with different cloud storage(S3, Azure Blob Storage, GCS). [polaris]
Posted by "mj006648 (via GitHub)" <gi...@apache.org>.
mj006648 commented on issue #1325:
URL: https://github.com/apache/polaris/issues/1325#issuecomment-4448535662
Thanks @flyrain for the quick guidance. The system-admin end-to-end perspective makes sense — I'll structure the docs that way for both backends.
Plan:
- Per-backend pages (Azure Blob + GCS), each covering:
1. Polaris server-side `STORAGE_CONFIG_INFO` setup
2. Client-side dependencies and credentials (Spark / Trino / PyIceberg)
3. End-to-end verification (create namespace + table + read back)
- Mirrors the structure of the existing S3 / Spark docs from #1435.
I'll keep skill.md as a follow-up once the docs land. Will open a draft PR soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document how Polaris works with different cloud storage(S3, Azure Blob Storage, GCS). [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on issue #1325:
URL: https://github.com/apache/polaris/issues/1325#issuecomment-2852427548
Per offline discussion with @adnanhemani, we will still need more doc on how different storage setup works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document how Polaris works with different cloud storage(S3, Azure Blob Storage, GCS). [polaris]
Posted by "mj006648 (via GitHub)" <gi...@apache.org>.
mj006648 commented on issue #1325:
URL: https://github.com/apache/polaris/issues/1325#issuecomment-4447111795
Hi @flyrain, following up on your note that more docs are needed for storage backends beyond what landed in #1435. If no one is currently working on the Azure Blob Storage and GCS sides, I'd like to pick those up.
Tentative plan, mirroring the structure of the existing S3 / Spark docs:
- Azure Blob Storage configuration page
- `STORAGE_CONFIG_INFO` example (tenant id, client id, secret vs. Workload Identity)
- ADLS Gen2 endpoint conventions
- Vended Credentials behavior (cross-ref to #418 if still relevant)
- Google Cloud Storage configuration page
- service-account-key vs. ADC vs. HMAC
- uniform vs. bucket-level IAM notes
- A short section on the client-side dependencies each backend needs (hadoop-azure, gcs-connector, etc.) when used via Spark / Trino / PyIceberg.
Does this match what you had in mind, or would you prefer a single consolidated "storage backends" page instead of per-backend pages? Happy to follow whichever structure fits the existing docs better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org