You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by Dennis Huo <hu...@gmail.com> on 2025/07/17 19:47:17 UTC
[DISCUSS] Quick sketch of next-steps to implement table/namespace-level RBAC and credential vending for Polaris Federation
Hey all,
I sketched out some design details for next-step features in Polaris
Federation based on discussion with Pooja, Rulin, Harish, and Teja:
https://docs.google.com/document/d/19Vm0Uy-EyEYtgd2-bZpPICODOB3rYQJvHeqL1ODOOLc/edit?tab=t.0
At a high level, currently the REST-based Federation MVP basically only
does two things:
- Catalog-level RBAC
- Simple passthrough to the remote REST Catalog
Per the original federation proposal, the responsibilities of a catalog
federation layer ultimately go beyond "simple passthrough", and
higher-level features entail making the federation layer actually an
"active" translation layer.
The three features outlined in the document are closely interrelated and
also serve as prerequisites for future plans:
- Credential-vending using the Polaris StorageConfig
- Polaris fetching the TableMetadata (and potentially other
metadata/manifest files) - particularly applicable if the remote catalog is
an implementation that only returns a metadata filename instead of
providing inline TableMetadata JSON
- Namespace/Table-level RBAC
In a nutshell, the main technical work for these next steps is to refactor
the remote-catalog client to live under a Polaris "wrapper" (referred to as
the "decorating delegator catalog" in the document) which formalizes the
responsibilities of this federation in terms of things like
credential-vending, construction of FileIO via FileIOFactory, etc.
Any input is welcome.
Cheers,
Dennis
Re: [DISCUSS] Quick sketch of next-steps to implement table/namespace-level RBAC and credential vending for Polaris Federation
Posted by Dennis Huo <hu...@gmail.com>.
Thanks for the feedback in the doc!
Yes, totally agree about refactoring to reduce the many conditional chains
and "instanceof" calls. This could be a good opportunity to aim for that
refactoring while adding these features.
On Fri, Jul 18, 2025 at 2:59 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi Dennis,
>
> Thanks for sharing this implementation plan. I left a couple of comments on
> the doc.
>
> I'm not sure I have a clear picture on what exactly you're planning to do,
> but it looks like a reasonable approach to me overall.
>
> I think refactoring AccessConfig to be a more prominent and self-sufficient
> "service" inside Polaris (shared for different use cases) makes a lot of
> sense.
>
> If possible, I wish the refactoring could reduce the amount of
> if/instanceof code in Polaris. I do not have any specific suggestions ATM,
> but I am willing to collaborate on PRs. To clarify: I mean
> if/elseif/elseif/... sequences for different logic per sub-type. I hope
> those could be converted to overridable (virtual) method calls (as an
> example).
>
> Cheers,
> Dmitri.
>
> On Thu, Jul 17, 2025 at 3:48 PM Dennis Huo <hu...@gmail.com> wrote:
>
> > Hey all,
> >
> > I sketched out some design details for next-step features in Polaris
> > Federation based on discussion with Pooja, Rulin, Harish, and Teja:
> >
> >
> https://docs.google.com/document/d/19Vm0Uy-EyEYtgd2-bZpPICODOB3rYQJvHeqL1ODOOLc/edit?tab=t.0
> >
> > At a high level, currently the REST-based Federation MVP basically only
> > does two things:
> >
> > - Catalog-level RBAC
> > - Simple passthrough to the remote REST Catalog
> >
> > Per the original federation proposal, the responsibilities of a catalog
> > federation layer ultimately go beyond "simple passthrough", and
> > higher-level features entail making the federation layer actually an
> > "active" translation layer.
> >
> > The three features outlined in the document are closely interrelated and
> > also serve as prerequisites for future plans:
> >
> > - Credential-vending using the Polaris StorageConfig
> > - Polaris fetching the TableMetadata (and potentially other
> > metadata/manifest files) - particularly applicable if the remote
> > catalog is
> > an implementation that only returns a metadata filename instead of
> > providing inline TableMetadata JSON
> > - Namespace/Table-level RBAC
> >
> >
> > In a nutshell, the main technical work for these next steps is to
> refactor
> > the remote-catalog client to live under a Polaris "wrapper" (referred to
> as
> > the "decorating delegator catalog" in the document) which formalizes the
> > responsibilities of this federation in terms of things like
> > credential-vending, construction of FileIO via FileIOFactory, etc.
> >
> > Any input is welcome.
> >
> > Cheers,
> > Dennis
> >
>
Re: [DISCUSS] Quick sketch of next-steps to implement table/namespace-level RBAC and credential vending for Polaris Federation
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Dennis,
Thanks for sharing this implementation plan. I left a couple of comments on
the doc.
I'm not sure I have a clear picture on what exactly you're planning to do,
but it looks like a reasonable approach to me overall.
I think refactoring AccessConfig to be a more prominent and self-sufficient
"service" inside Polaris (shared for different use cases) makes a lot of
sense.
If possible, I wish the refactoring could reduce the amount of
if/instanceof code in Polaris. I do not have any specific suggestions ATM,
but I am willing to collaborate on PRs. To clarify: I mean
if/elseif/elseif/... sequences for different logic per sub-type. I hope
those could be converted to overridable (virtual) method calls (as an
example).
Cheers,
Dmitri.
On Thu, Jul 17, 2025 at 3:48 PM Dennis Huo <hu...@gmail.com> wrote:
> Hey all,
>
> I sketched out some design details for next-step features in Polaris
> Federation based on discussion with Pooja, Rulin, Harish, and Teja:
>
> https://docs.google.com/document/d/19Vm0Uy-EyEYtgd2-bZpPICODOB3rYQJvHeqL1ODOOLc/edit?tab=t.0
>
> At a high level, currently the REST-based Federation MVP basically only
> does two things:
>
> - Catalog-level RBAC
> - Simple passthrough to the remote REST Catalog
>
> Per the original federation proposal, the responsibilities of a catalog
> federation layer ultimately go beyond "simple passthrough", and
> higher-level features entail making the federation layer actually an
> "active" translation layer.
>
> The three features outlined in the document are closely interrelated and
> also serve as prerequisites for future plans:
>
> - Credential-vending using the Polaris StorageConfig
> - Polaris fetching the TableMetadata (and potentially other
> metadata/manifest files) - particularly applicable if the remote
> catalog is
> an implementation that only returns a metadata filename instead of
> providing inline TableMetadata JSON
> - Namespace/Table-level RBAC
>
>
> In a nutshell, the main technical work for these next steps is to refactor
> the remote-catalog client to live under a Polaris "wrapper" (referred to as
> the "decorating delegator catalog" in the document) which formalizes the
> responsibilities of this federation in terms of things like
> credential-vending, construction of FileIO via FileIOFactory, etc.
>
> Any input is welcome.
>
> Cheers,
> Dennis
>