You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "rdsarvar (via GitHub)" <gi...@apache.org> on 2025/08/11 18:13:25 UTC
[PR] feat: adding unauthenticated HDFS storage support to catalogs (#85) [polaris]
rdsarvar opened a new pull request, #2322:
URL: https://github.com/apache/polaris/pull/2322
# TODO - Description
Initially contains per catalog resources and username configurations that enable Polaris to configure and impersonate against an HDFS cluster
<!--
Possible security vulnerabilities: STOP here and contact security@apache.org instead!
Please update the title of the PR with a meaningful message - do not leave it "empty" or "generated"
Please update this summary field:
The summary should cover these topics, if applicable:
* the motivation for the change
* a description of the status quo, for example the current behavior
* the desired behavior
* etc
PR checklist:
- Do a self-review of your code before opening a pull request
- Make sure that there's good test coverage for the changes included in this PR
- Run tests locally before pushing a PR (./gradlew check)
- Code should have comments where applicable. Particularly hard-to-understand
areas deserve good in-line documentation.
- Include changes and enhancements to the documentation (in site/content/in-dev/unreleased)
- For Work In Progress Pull Requests, please use the Draft PR feature.
Make sure to add the information BELOW this comment.
Everything in this comment will NOT be added to the PR description.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: adding unauthenticated HDFS storage support to catalogs (#85) [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on PR #2322:
URL: https://github.com/apache/polaris/pull/2322#issuecomment-3178914700
We have been (more precisely: are) discussing support for Hive, and implicitly HDFS, in Polaris.
There are a couple of concerns about adding (more) Hadoop stuff to Polaris.
Hadoop types like `org.apache.hadoop.security.UserGroupInformation` et al rely on functionality that has already been deprecated since Java 17 ([JEP-411](https://openjdk.org/jeps/411)) and is already _removed_ in Java since version 24 ([JEP-486](https://openjdk.org/jeps/486))). Although we technically do not need 24 (or the upcoming Java 25) yet, we might have to.
As we want to support these use cases, the approach would be to have a Polaris instance that _federates_ to an instance, which can then have have hard "max Java version" restrictions but also do other "restrictive" things like Kerberos.
This all falls into the "catalog federation" bucket. We'd appreciate your input on the dev-mailing-list and in-person discussions in our biweekly community sync!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: adding unauthenticated HDFS storage support to catalogs (#85) [polaris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2322:
URL: https://github.com/apache/polaris/pull/2322#issuecomment-3283374128
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: adding unauthenticated HDFS storage support to catalogs (#85) [polaris]
Posted by "rdsarvar (via GitHub)" <gi...@apache.org>.
rdsarvar commented on PR #2322:
URL: https://github.com/apache/polaris/pull/2322#issuecomment-3178989292
> We have been (more precisely: are) discussing support for Hive, and implicitly HDFS, in Polaris. There are a couple of concerns about adding (more) Hadoop stuff to Polaris.
>
> Hadoop types like `org.apache.hadoop.security.UserGroupInformation` et al rely on functionality that has already been deprecated since Java 17 ([JEP-411](https://openjdk.org/jeps/411)) and is already _removed_ in Java since version 24 ([JEP-486](https://openjdk.org/jeps/486))). Although we technically do not need 24 (or the upcoming Java 25) yet, we might have to.
>
> As we want to support these use cases, the approach would be to have a Polaris instance that _federates_ to an instance, which can then have have hard "max Java version" restrictions but also do other "restrictive" things like Kerberos.
>
> This all falls into the "catalog federation" bucket. We'd appreciate your input on the dev-mailing-list and in-person discussions in our biweekly community sync!
Hi @snazy thanks for the response --
Interesting solution, so just to recap to make sure I'm on the same page the thought is to run a separate sidecar / service which itself can be pegged to a lower Java version and support any HDFS related operations there? So in essence the workflow would now look like:
1. User makes request to Polaris catalog (ex: new table)
2. Polaris determines the storage type is HDFS and proxies the request to the HDFS catalog service (REST API I assume)
3. The HDFS catalog service would perform required actions on behalf of Polaris, returning what happened through structured JSON response
4. Polaris would respond to the user and make any required persistence changes
Thus decoupling the potentially unsafe (or lesser version) Hadoop usage to users that are explicitly requiring it.
I'll join the Slack workspace, do you have reference to the channel (or thread) that this is being discussed? Or is it mostly in the syncs / mailing list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: adding unauthenticated HDFS storage support to catalogs (#85) [polaris]
Posted by "rdsarvar (via GitHub)" <gi...@apache.org>.
rdsarvar commented on PR #2322:
URL: https://github.com/apache/polaris/pull/2322#issuecomment-3179040176
Sounds good, for reference of anyone coming into this PR it can be found here: https://lists.apache.org/thread/gykvrd103hyxy6b829th45lo0qtb78mn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: adding unauthenticated HDFS storage support to catalogs (#85) [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on PR #2322:
URL: https://github.com/apache/polaris/pull/2322#issuecomment-3179024880
That's (pretty much) correct.
But let's keep the discussion on the dev-ML.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: adding unauthenticated HDFS storage support to catalogs (#85) [polaris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2322: feat: adding unauthenticated HDFS storage support to catalogs (#85)
URL: https://github.com/apache/polaris/pull/2322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org