Re: Context-Aware Functions for Apache Polaris

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi guys

About FGAC, I think we have to be more pragmatic.

We can identify three parts in FGAC:
- column masking
- column filtering
- row filtering

I believe the easiest to start with is column masking.

Let's start with column masking. The cinematic would be:
1. The query engine calls the catalog getTable() with credentials
2. The catalog calls the authorizer to verify credentials and to get
the associated roles
3. The catalog returns a "mask function name" to the query engine
(basically an id). This function name should be part of the Iceberg
REST getTable response (it's not today, so maybe we should start a
discussion with the Iceberg community to add this optional "mask"
field.
4. Then the query engine "maps" the "mask name" to its logic.

It means that:
1. The query engine is "coupled" with Polaris "mask name": it should
"translate" the mask name to its internal function. If the "mask name"
is not understood by the query engine, a "permission exception"
happens.
2. With this approach, we don't need an abstract representation in
Polaris (no Substrait, no Coral, ...). Polaris just "stores" mask
names.
3. It's a PoC approach that would allow us to evaluate the "plumbing"
required between catalog/authorizer/query engine.

Thoughts ?

I propose to start drafting a document with step by step FGAC patterns.

Regards
JB

On Tue, May 20, 2025 at 10:05 AM Prashant Singh
<pr...@snowflake.com.invalid> wrote:
>
> > Let's work together on a proper design.
>
> For sure ! will be reaching out soon for this !
>
> Best,
> Prashant Singh

Re: Context-Aware Functions for Apache Polaris

[Prashant Singh <pr...@snowflake.com.INVALID>]

Hey Alex,
Thank you for your interest in the thread, and appreciate your inputs !
It seems like there is a misunderstanding about the thread, my intention
was just to introduce these catalog evaluated functions or *variables*,
which are evaluated based on callers context, and are replaced inline, they
will have a fixed type, also these were just meant to be something to be
evaluated by Polaris, so engine doesn't have to use or introduce, that's
true that context functions can be used in FGAC, FGAC was not what i was
proposing (I have proposal for it too, which I am thinking of sharing with
the wider community). If you will see, the *context_aware* functions are
something which unity-catalog supports without FGAC as well, like I shared
in the thread, so does snowflake and that's the whole case I was trying to
make,

Re: FGAC policies and E2E FGAC experience
Yes, we need to start with FGAC policies first to agree with the type of
policies and only then think of an enforcement model !

Re: Context functions not as UDF
Yes, that's what I have tried to state in the thread too. as unlike UDF's
this function doesn't even have a body and to be specific they can very
well be called context variables instead.

Re: a few common, constant-foldable expressions in FGAC policies, and go
from there.
Precisely, let me share the doc (by next week) which we have been
interating on internally for a while now, I hope all this makes sense once
shared.

Best,
Prashant Singh


On Tue, Jun 17, 2025 at 5:48 AM Alex Dutra <al...@dremio.com.invalid>
wrote:

> Hi all,
>
> Catching up quite late with this discussion thread, my apologies.
> Trying to sum up my impressions so far:
>
> We should definitely avoid parsing SQL and altering view definitions.
> Instead it seems that Eric's proposal of defining FGAC policies with a
> clear syntax and spec is a much more robust approach.
>
> We definitely should focus initially on the 3 main purposes of FGAC
> policies: row filtering, column filtering and column masking.
>
> Prashant's initial proposal seems to hint at the fact that some
> function calls in FGAC policies could be constant-folded server-side.
> E.g. the expression "IF has_role('admin')" in an FGAC policy body
> could be inlined to FALSE by the server, depending on the current
> user. This is probably what Eric meant by "catalog-driven predicates".
>
> But not all function calls will be constant-foldable, and some of them
> will have to be interpreted on the engine side (or, as Eric puts it,
> "engine-driven predicates"). Then the question is: how would an engine
> be able to incorporate a function/predicate like "has_role('admin')"
> into the query plan?
>
> That is imho a very complex topic. It probably requires the FGAC
> specification to settle on a set of "standard" or "built-in" functions
> that all REST catalogs must support. But, such functions cannot be
> pure, as the function result also depends on the context, not only on
> the function arguments (IOW: has_role('admin') can return TRUE or
> FALSE). Thus, I'm not sure we can model these functions as (scalar)
> UDFs, since for now the UDF specification defines scalar UDFs as
> "based on predefined logic". We might need a different concept here.
>
> That said, we could maybe start working on FGAC with support for just
> a few common, constant-foldable expressions in FGAC policies, and go
> from there.
>
> My 2c,
>
> Alex
>
> On Sat, Jun 7, 2025 at 12:34 AM Prashant Singh
> <pr...@snowflake.com.invalid> wrote:
> >
> > In terms of achieving this I was thinking of this as the following as
> well
> > :
> > 1. role based column filtering as -> This should be achievable, without a
> > lot of lift shift.
> > 2. simple row based filters
> > 3. column mask and row mask (when Iceberg UDF are established)
> > And agree we just need to store the mask name and we can send it back to
> > the engine in many ways,  but i think for mask to mean something for all
> > engines we need Iceberg UDF's for example email_mask means XYZ ....
> across
> > Engine A, Engine B, how we achieve this if by IR or just storing it it in
> > its individual dialect that still need some brainstorming, but I think in
> > principle engine should come back to catalog for getting the function
> > definition would imho be the best bet ! So maybe tackling the row mask
> and
> > column mask when UDF are first class citizens might be best ,Though
> > starting with 1 seems the simplest !
> > Given that I am working on a proposal doc for incorporating these
> thoughts
> > for Polaris, I will share with the community *soon* and would love to get
> > all of your feedback  !
> >
> > Best,
> > Prashant Singh
> >
> >
> > On Fri, Jun 6, 2025 at 2:48 PM Eric Maynard <er...@gmail.com>
> > wrote:
> >
> > > It seems to me that the *easiest* to start with would be role-based
> column
> > > filtering. There are no functions to grapple with, no dialect
> differences.
> > > You simply grab the list of columns that a given principal role has
> access
> > > to according to the FGAC policy attached to a given table.
> > >
> > > --EM
> > >
>

Re: Context-Aware Functions for Apache Polaris

[Alex Dutra <al...@dremio.com.INVALID>]

Hi all,

Catching up quite late with this discussion thread, my apologies.
Trying to sum up my impressions so far:

We should definitely avoid parsing SQL and altering view definitions.
Instead it seems that Eric's proposal of defining FGAC policies with a
clear syntax and spec is a much more robust approach.

We definitely should focus initially on the 3 main purposes of FGAC
policies: row filtering, column filtering and column masking.

Prashant's initial proposal seems to hint at the fact that some
function calls in FGAC policies could be constant-folded server-side.
E.g. the expression "IF has_role('admin')" in an FGAC policy body
could be inlined to FALSE by the server, depending on the current
user. This is probably what Eric meant by "catalog-driven predicates".

But not all function calls will be constant-foldable, and some of them
will have to be interpreted on the engine side (or, as Eric puts it,
"engine-driven predicates"). Then the question is: how would an engine
be able to incorporate a function/predicate like "has_role('admin')"
into the query plan?

That is imho a very complex topic. It probably requires the FGAC
specification to settle on a set of "standard" or "built-in" functions
that all REST catalogs must support. But, such functions cannot be
pure, as the function result also depends on the context, not only on
the function arguments (IOW: has_role('admin') can return TRUE or
FALSE). Thus, I'm not sure we can model these functions as (scalar)
UDFs, since for now the UDF specification defines scalar UDFs as
"based on predefined logic". We might need a different concept here.

That said, we could maybe start working on FGAC with support for just
a few common, constant-foldable expressions in FGAC policies, and go
from there.

My 2c,

Alex

On Sat, Jun 7, 2025 at 12:34 AM Prashant Singh
<pr...@snowflake.com.invalid> wrote:
>
> In terms of achieving this I was thinking of this as the following as well
> :
> 1. role based column filtering as -> This should be achievable, without a
> lot of lift shift.
> 2. simple row based filters
> 3. column mask and row mask (when Iceberg UDF are established)
> And agree we just need to store the mask name and we can send it back to
> the engine in many ways,  but i think for mask to mean something for all
> engines we need Iceberg UDF's for example email_mask means XYZ .... across
> Engine A, Engine B, how we achieve this if by IR or just storing it it in
> its individual dialect that still need some brainstorming, but I think in
> principle engine should come back to catalog for getting the function
> definition would imho be the best bet ! So maybe tackling the row mask and
> column mask when UDF are first class citizens might be best ,Though
> starting with 1 seems the simplest !
> Given that I am working on a proposal doc for incorporating these thoughts
> for Polaris, I will share with the community *soon* and would love to get
> all of your feedback  !
>
> Best,
> Prashant Singh
>
>
> On Fri, Jun 6, 2025 at 2:48 PM Eric Maynard <er...@gmail.com>
> wrote:
>
> > It seems to me that the *easiest* to start with would be role-based column
> > filtering. There are no functions to grapple with, no dialect differences.
> > You simply grab the list of columns that a given principal role has access
> > to according to the FGAC policy attached to a given table.
> >
> > --EM
> >

Re: Context-Aware Functions for Apache Polaris

[Prashant Singh <pr...@snowflake.com.INVALID>]

In terms of achieving this I was thinking of this as the following as well
:
1. role based column filtering as -> This should be achievable, without a
lot of lift shift.
2. simple row based filters
3. column mask and row mask (when Iceberg UDF are established)
And agree we just need to store the mask name and we can send it back to
the engine in many ways,  but i think for mask to mean something for all
engines we need Iceberg UDF's for example email_mask means XYZ .... across
Engine A, Engine B, how we achieve this if by IR or just storing it it in
its individual dialect that still need some brainstorming, but I think in
principle engine should come back to catalog for getting the function
definition would imho be the best bet ! So maybe tackling the row mask and
column mask when UDF are first class citizens might be best ,Though
starting with 1 seems the simplest !
Given that I am working on a proposal doc for incorporating these thoughts
for Polaris, I will share with the community *soon* and would love to get
all of your feedback  !

Best,
Prashant Singh


On Fri, Jun 6, 2025 at 2:48 PM Eric Maynard <er...@gmail.com>
wrote:

> It seems to me that the *easiest* to start with would be role-based column
> filtering. There are no functions to grapple with, no dialect differences.
> You simply grab the list of columns that a given principal role has access
> to according to the FGAC policy attached to a given table.
>
> --EM
>

Re: Context-Aware Functions for Apache Polaris

[Eric Maynard <er...@gmail.com>]

It seems to me that the *easiest* to start with would be role-based column
filtering. There are no functions to grapple with, no dialect differences.
You simply grab the list of columns that a given principal role has access
to according to the FGAC policy attached to a given table.

--EM