You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by Dennis Huo <dh...@apache.org> on 2026/05/28 05:46:21 UTC
[DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Hi All,
One big emerging enterprise use case coming up as more people consolidate
Data Lakehouses and Catalogs is something commonly known as "Data Sharing",
an more specifically over the course of adoption of open table formats
"Open Data Sharing".
Examples of existing managed service providers' Data Sharing features:
https://www.databricks.com/product/delta-sharing
https://docs.snowflake.com/en/user-guide/data-sharing-intro
https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
The basic idea is that when you share data between different companies, you
need a first-class governance/management layer and extra bells-and-whistles
that are distinct from just the basic capabilities of RBAC or generalized
access-control (i.e. if you're sharing across partially-untrusted org
boundaries, you don't just let the consumer organization log into your
datalake like one of your own employees).
JB and I put together this high-level proposal for supporting Open Sharing
in Polaris:
https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
Tentatively, it means adding ~5 logical data model constructs, some of
which may be a first-class PolarisEntity type, others subtypes of existing
entities, and others just a nested construct:
- ShareEntity (would behave similarly to a Catalog)
- ExternalConsumer (mostly inherits from Principal)
- Listing (Similar to a "role grant" but has different metadata)
- EndpointConfig (nested config under Listing)
- ShareMembership (Similar to a "securable grant" but different metadata)
Feedback/comments welcome! I'll also bring it up for live discussion if
there's time in the community sync.
Cheers,
Dennis
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Dennis,
Interesting proposal!
I left a few comments in the doc.
At a high level, from my POV, I believe we need to clarify the high-level
requirements. I do not insist on using the requirements approach, actually,
but the doc already uses that terminology. Alternatively, we could focus on
the use cases Open Sharing enables. In any case, I believe it would be
beneficial to draw a clean line between what we would like to achieve and
how we achieve that.
I tend to think we need to distinguish Consumption aspects from Governance
aspects. The former will generally be an Engine-side concern, while the
latter will be the data owner's concern. Different actors will be
involved on both sides of this line.
On the Consumption side, using IRC does seem like the best solution
nowadays.
On the Governance side, I believe some aspects of this proposal can benefit
from more discussion. In particular:
* It is not clear whether / how the Polaris Authorizer is involved on the
"shared" access path. Are new authorizable operations expected to be added
to the Authorizer SPI?
* It is not clear whether all Share users (external) have the same or
different access rights to the shared tables.
* How external IdPs can be integrated. The doc mentions OIDC_FEDERATION,
but it's not clear how it relates to ExternalConsumer. Where / how is OIDC
federation configured?
* It would be nice to delineate Authentication aspects from Authorization
aspects in Shares/Listings. I hope this will make it easier to evolve the
system by adding different AuthN methods later.
* In the case of AWS S3 what controls the role (plus External ID, etc.)
used for minting STS session (vended) credentials? Are they supposed to be
the same or different for different external clients?
* If a view is shared, how can we be sure its SQL definition is resolvable
(in terms of following table names/paths) in the context of a share?
Thanks,
Dmitri.
On Thu, May 28, 2026 at 1:46 AM Dennis Huo <dh...@apache.org> wrote:
> Hi All,
>
> One big emerging enterprise use case coming up as more people consolidate
> Data Lakehouses and Catalogs is something commonly known as "Data Sharing",
> an more specifically over the course of adoption of open table formats
> "Open Data Sharing".
>
> Examples of existing managed service providers' Data Sharing features:
>
> https://www.databricks.com/product/delta-sharing
> https://docs.snowflake.com/en/user-guide/data-sharing-intro
> https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
>
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
>
> The basic idea is that when you share data between different companies, you
> need a first-class governance/management layer and extra bells-and-whistles
> that are distinct from just the basic capabilities of RBAC or generalized
> access-control (i.e. if you're sharing across partially-untrusted org
> boundaries, you don't just let the consumer organization log into your
> datalake like one of your own employees).
>
> JB and I put together this high-level proposal for supporting Open Sharing
> in Polaris:
>
>
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
>
> Tentatively, it means adding ~5 logical data model constructs, some of
> which may be a first-class PolarisEntity type, others subtypes of existing
> entities, and others just a nested construct:
>
> - ShareEntity (would behave similarly to a Catalog)
> - ExternalConsumer (mostly inherits from Principal)
> - Listing (Similar to a "role grant" but has different metadata)
> - EndpointConfig (nested config under Listing)
> - ShareMembership (Similar to a "securable grant" but different
> metadata)
>
> Feedback/comments welcome! I'll also bring it up for live discussion if
> there's time in the community sync.
>
> Cheers,
> Dennis
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dennis Huo <dh...@apache.org>.
Makes sense, thanks! I've gone ahead with the new split in the PR:
1. polaris-shares-api.yaml - Shares and ShareMembership for adding
securables to the Share
2. polaris-shares-consumer-management-api.yaml -
Consumer/Listing/Credentials management
We can make sure the SPIs allow the core Shares part to operate
independently from the consumer-management side.
On Thu, Oct 22, 2026 at 4:27 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi Dennis,
>
> Thanks for the detailed path forward.
>
> The stages / compartmentalized approach make sense to me.
>
> 1) The Shares API defines "what to share". It does not depend on anything
> else.
>
> 2) The Authorizer SPI changes provide a way to express Share access
> requests.
>
> 3) Internal RBAC implements (2) on top of the internal Principals and
> Roles.
>
> I suppose ExternalConsumer will fit here.
>
> 4) External Authorizer implementations adapt to (2) and allow users to
> configure access rules in their respective configuration styles.
>
> Does that make sense?
>
> Thanks,
> Dmitri.
>
> On Thu, Oct 22, 2026 at 6:58 PM Dennis Huo <dh...@apache.org> wrote:
>
> > Right, sorry, to clarify, my stated bullets in the last email were my
> > hypothetical portrayal of what your proposed core set of APIs would be,
> not
> > what my current proposal says (so that you can confirm or deny whether
> > those three bullets convey your ideal state).
> >
> > It sounds like (2) and (3) match your intent, and (1) is exactly where
> the
> > current spec PR differs, since it contains consumers and listings.
> >
> > For the particular use cases that went into the scoping for the Polaris
> > Shares proposal, the ExternalConsumer is fairly fundamental to extracting
> > as a first-class object.
> >
> > But you're right that there could be a trimmed scoping that allows
> > implementors to fully abstract away the ExternalConsumer/Listing
> semantics
> > instead. In the same way, there's even a fully-trimmed formulation where
> we
> > don't implement *any* management APIs at all: Where "Sharing" is just
> data
> > providers providing dedicated lightweight Polaris servers, or adding a
> very
> > simple config-file-based mapping for what data to make accessible to
> > dedicated "sharing" Principals or PrincipalRoles.
> >
> > We could reconcile this by also moving the ExternalConsumer/Listings APIs
> > to the same "optional" yaml file I moved the other
> > ConsumerCredential-related endpoints into and keeping the "core Shares
> API"
> > limited to what you wanted. Broadly, the two pillars of the proposal are:
> >
> > 1. *What* to share
> > 2. *With whom* to share
> >
> > The core Shares API would only cover the first, in line with your
> proposal:
> > CRUD on Shares and the securables contained in them (listShares,
> > createShare, getShare, updateShare, deleteShare, listShareMembers,
> > updateShareMembers).
> >
> > The optional file would cover the second: ExternalConsumers, Listings,
> the
> > endpoint configuration, and credentials for the polaris-standalone case.
> >
> > Polaris service-runners could then decide if they just want sharing
> > consumers to live in the exact same identity domain as their other native
> > Polaris readers, or segregate them by enabling the "With whom" half of
> the
> > API.
> >
> > One thing to discuss for the "core-only" case - today the management API
> > doesn't syntactically support granting a share to a PrincipalRole, so
> we'd
> > need to extend grant types or expose some of the internals of a share's
> > backing catalog role. I guess we can iteratively refine any necessary API
> > changes as we discover these in the core-only implementation.
> >
> > Note that the "with whom" half is necessary for the full "segregation"
> > reqiurements stated in the original proposal doc, such as preventing
> > assignment to other roles or accessing the "normal" catalog endpoints,
> > which is why it can't be *fully* subsumed by only the native Polaris
> > management API.
> >
> > It also doesn't conflict with external IdPs: Compare to Databricks'
> > RECIPIENT for example, which can wrap a Databricks identity, a
> > provider-issued token, or the recipients own IdP through an OIDC
> federatoin
> > policy, and shares are granted uniformly to it in all three cases:
> >
> https://docs.databricks.com/aws/en/delta-sharing/create-recipient-oidc-fed
> >
> > Even in the full API model my PR proposes, we could let the Listing be
> > something the authorizer looks at alongside other authz conext, so the
> > pluggable authorizers/OPA/Ranger/etc can integrate as the authoritative
> > layer.
> >
> >
> > On Thu, Oct 22, 2026 at 2:00 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Dennis,
> > >
> > > > - Shares API defines only shares and the objects placed in shares,
> with
> > > no
> > > > Consumer or Listing
> > >
> > > POST /shares/{shareName}/listings taking a CreateListingRequest with a
> > > `consumerName` is still part of `spec/polaris-shares-api.yaml`.
> > >
> > > I'm kind of confused about the scope of "shares" API vs. other APIs :)
> > > Where are the boundaries?
> > >
> > > Would could be the minimal isoalated API sufficient for managing
> Shares?
> > >
> > > > - Controlling who can read shares is only ordinary Polaris authz,
> e.g.
> > > > assignPrincipalRoletoPrincipal or
> > > addGrantOnSecurableToPrincipalRole(share,
> > > > principalRole)
> > >
> > > Sounds good.
> > >
> > > > - The partner organization consuming the share is just a principal,
> > which
> > > > could either be Polaris-managed or from an external IdP
> > >
> > > This also sounds good to me.
> > >
> > > I guess the only concern / confusion area is around Listing / Consumer
> > > (first point above).
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > >
> > > On Thu, Oct 22, 2026 at 2:07 PM Dennis Huo <dh...@apache.org> wrote:
> > >
> > > > Thanks Dmitri, that does help highlight the the concern. Just to make
> > > sure
> > > > I understand what you have in mind:
> > > >
> > > > - Shares API defines only shares and the objects placed in shares,
> with
> > > no
> > > > Consumer or Listing
> > > > - Controlling who can read shares is only ordinary Polaris authz,
> e.g.
> > > > assignPrincipalRoletoPrincipal or
> > > addGrantOnSecurableToPrincipalRole(share,
> > > > principalRole)
> > > > - The partner organization consuming the share is just a principal,
> > which
> > > > could either be Polaris-managed or from an external IdP
> > > >
> > > > Is that correct?
> > > >
> > > > On Wed, Oct 21, 2026 at 4:34 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Dennis,
> > > > >
> > > > > Let me try to restate my concern from another angle. Apologies
> that I
> > > was
> > > > > not very clear in my previous messages.
> > > > >
> > > > > Currently the create listing doc says: "Create a listing, which
> > grants
> > > an
> > > > > external consumer access to this share".
> > > > >
> > > > > So, this binds authorization decisions about a Share (Listing) to
> > > > > ExternalConsumer.
> > > > >
> > > > > In IRC use cases a PolarisPrincipal is established by
> authenticating
> > > the
> > > > > client request (either via internal JWT or via external IdP),
> > > > Subsequently,
> > > > > the PolarisPrincipal is passed to PolarisAuthorizer for making
> > > > > authorization decisions. The latter part can be external too (e.g.
> > OPA
> > > /
> > > > > Ranger).
> > > > >
> > > > > (Side note: the "external" work in ExternalConsumer is actually
> quite
> > > > > confusing with respect to prior usage of that word in external IdP
> > use
> > > > > cases.)
> > > > >
> > > > > In my view, the ExternalConsumer and related concepts in current
> > Shares
> > > > API
> > > > > do not align with the external IdP and Authorizer workflows.
> > > > >
> > > > > IIRC, actual access to shared tables is going to use the IRC API,
> > > right?
> > > > > That access path in existing Polaris use cases maps well to
> external
> > > IdP
> > > > > and external authorizers.
> > > > >
> > > > > Why does the Shares API need to define a "Consumer" as a separate
> > > entity
> > > > > and narrow authorization decisions to its relationship with
> Listings?
> > > > >
> > > > > I believe authentication and authorization options should remain as
> > > wide
> > > > as
> > > > > possible because these matters are usually addressed very
> differently
> > > in
> > > > > each deployment.
> > > > >
> > > > > To be clear: I do not mind defining Share "Consumers" in the
> internal
> > > > > Polaris RBAC domain. It can fit well there, I'm only suggesting
> > keeping
> > > > the
> > > > > Shares API itself neutral to Auth N/Z concerns.
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Wed, Oct 21, 2026 at 7:02 PM Dennis Huo <dh...@apache.org>
> wrote:
> > > > >
> > > > > > Ah thanks for clarifying - to be clear, the ConsumerCredential is
> > > *not*
> > > > > > strictly necessary for the Listing in terms of spec
> expressibility,
> > > > only
> > > > > > the ExternalConsumer is.
> > > > > >
> > > > > > An ExternalConsumer doesn't need to be backed by a
> > > ConsumerCredential;
> > > > > the
> > > > > > ExternalConsumer is an abstract Sharing-specific pointer to
> > whatever
> > > > does
> > > > > > the indirection in the sharing implementation.
> > > > > >
> > > > > > Responded to your github comment - I'll split anything that
> touches
> > > > > > "ConsumerCredential" into a separate yaml file so it's easier to
> > see
> > > > the
> > > > > > core auth-agnostic sharing spec in one file, with the more
> > > > > > implementation-specific decorations around ConsumerCredential in
> an
> > > > > > auxiliary file.
> > > > > >
> > > > > > I suppose another option is to even split it into separate PRs
> (in
> > > > > addition
> > > > > > to separate yaml files?). Then we can easily see one commit
> that's
> > > just
> > > > > the
> > > > > > pure abstract auth-agnostic API, that in theory private Polaris
> > > > > > implementations could already take and implement on external-IdP
> > > > > > architectures to support the same agnostic spec.
> > > > > >
> > > > > > On Wed, Oct 21, 2026 at 1:54 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Dennis,
> > > > > > >
> > > > > > > While I still support the idea of splitting OpenAPI yaml files
> > and
> > > > URI
> > > > > > > bases between the Management and Sharing APIs, I do not think
> it
> > is
> > > > > > tightly
> > > > > > > coupled with my proposal for the authentication-neural Sharing
> > API
> > > > > spec.
> > > > > > >
> > > > > > > To give an example, currently [5446] CreateListingRequest
> > includes
> > > a
> > > > > > > "consumerName". This immediately binds the Listing definition
> to
> > > > > > > an ExternalConsumer and by extension to ConsumerCredential.
> > > > > > >
> > > > > > > ConsumerCredential is naturally part of the authentication
> > process
> > > > > during
> > > > > > > Share access. I believe ConsumerCredential (and related
> entities)
> > > is
> > > > > > worth
> > > > > > > isolating from the Shares API to increase interoperability.
> > > > > > >
> > > > > > > Implementations that choose to rely on ConsumerCredential for
> > Share
> > > > > > access
> > > > > > > can still do so. However, ConsumerCredential will not be a
> > > > requirement
> > > > > > for
> > > > > > > all possible Share API implementations.
> > > > > > >
> > > > > > > This can work with any OpenAPI yaml and URI paths layout.
> > > > > > >
> > > > > > > I'd like to resolve this aspect before we go back to URI paths
> > and
> > > > YAML
> > > > > > > file locations. Actually, I do not feel too strongly about path
> > > > > > prefixes. I
> > > > > > > do believe separate prefixes make implementations easier in
> many
> > > > > aspects
> > > > > > > (including network setup), but interleaving those paths with
> the
> > > > > > Management
> > > > > > > API is not a blocker, from my POV, only an inconvenience.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > [5446] https://github.com/apache/polaris/pull/5446
> > > > > > >
> > > > > > > On Wed, Oct 21, 2026 at 3:19 PM Dennis Huo <dh...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > > > Yeah, that's a good topic to discuss; I guess it touches on
> our
> > > > core
> > > > > > > > (unresolved?) conventions on how we organize API url-prefixes
> > and
> > > > > yaml
> > > > > > > > files. Reposting my response to the github comment as well:
> > > > > > > >
> > > > > > > > Right, I think we could certainly split the API routes and
> > > > > > > yaml-definitions
> > > > > > > > even further if we want to ensure a well-defined
> > auth-detail-free
> > > > > core,
> > > > > > > but
> > > > > > > > for the standalone-Polaris use case we'd still need to define
> > it
> > > > > > > > *somewhere*
> > > > > > > > under a Sharing-specific surface, right?
> > > > > > > >
> > > > > > > > @dimas-b <https://github.com/dimas-b> Were you thinking of
> > just
> > > > > > > splitting
> > > > > > > > the yaml file but keeping the share-management root, or
> adding
> > a
> > > > > third
> > > > > > > > sharing prefix like
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials...
> > > > > > > > ?
> > > > > > > >
> > > > > > > > Or just yaml-file split would be keeping
> > > > > > > >
> > > /api/shares-management/v1/external-consumers/{name}/credentials...
> > > > > but
> > > > > > > > putting it in a separate yaml file to make it easy for
> > > > > service-runners
> > > > > > to
> > > > > > > > choose not to support that whole yaml file's concrete APIs.
> > > > > > > >
> > > > > > > > On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > Re: the shating API PR [5446], I'm repeating my GH comment
> > here
> > > > for
> > > > > > > > > visibility.
> > > > > > > > >
> > > > > > > > > I'd like the API to be neutral from the authentication
> > > > perspective.
> > > > > > > > >
> > > > > > > > > This is how we handle the IRC spec in Polaris. The API
> > declares
> > > > > that
> > > > > > > > > clients need to authenticate, but specific details about
> > > > > principals,
> > > > > > > > > clients and secrets are delegated to the server.
> > > > > > > > >
> > > > > > > > > I think this is a sound approach for the sharing API too.
> > > > Specific
> > > > > > > > details
> > > > > > > > > related to native Polaris principals (ConsumerCredential,
> > etc.)
> > > > can
> > > > > > > > > probably remain in the Management API.
> > > > > > > > >
> > > > > > > > > WDYT?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > [5446] https://github.com/apache/polaris/pull/5446
> > > > > > > > >
> > > > > > > > > On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré <
> > > > > > jb@nanthrax.net>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi
> > > > > > > > > >
> > > > > > > > > > As I said during the community meeting, let's focus on
> the
> > > API
> > > > PR
> > > > > > > > > > first. We will build from there.
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > JB
> > > > > > > > > >
> > > > > > > > > > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <
> dhuo@apache.org
> > >
> > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > +1, thanks for the discussion everyone!
> > > > > > > > > > >
> > > > > > > > > > > Yes, I meant to flip it to "ready to review" earlier;
> > > flipped
> > > > > it
> > > > > > > now.
> > > > > > > > > I'm
> > > > > > > > > > > still working through the existing set of comments and
> > may
> > > > need
> > > > > > to
> > > > > > > > come
> > > > > > > > > > > back to it in the next few days before getting proper
> > > > replies,
> > > > > > but
> > > > > > > > feel
> > > > > > > > > > > free to add additional comments.
> > > > > > > > > > >
> > > > > > > > > > > Recapping the community sync discussion re: Dmitri's
> last
> > > > > > message,
> > > > > > > > I'm
> > > > > > > > > > > fully aligned on building the sharing on top of
> existing
> > > > authz
> > > > > > > > > > mechanisms,
> > > > > > > > > > > instead of being a new mechanism, and indeed external
> IdP
> > > is
> > > > > > > intended
> > > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > a first-class supported use case.
> > > > > > > > > > >
> > > > > > > > > > > The latest distinction is that the ExternalConsumer
> isn't
> > > > > > > > fundamentally
> > > > > > > > > > > identified by any internal PolarisPrincipal, but rather
> > > that
> > > > it
> > > > > > > > serves
> > > > > > > > > > as a
> > > > > > > > > > > wrapper that references some identity, whether that
> > > identity
> > > > is
> > > > > > > > defined
> > > > > > > > > > in
> > > > > > > > > > > an external IdP without any latent in-Polaris
> > > representation
> > > > or
> > > > > > if
> > > > > > > > > it's a
> > > > > > > > > > > hidden PolarisPrincipal (for the Polaris-standalone
> RBAC
> > > use
> > > > > > case).
> > > > > > > > > > >
> > > > > > > > > > > The sharing endpoint via ExternalConsumer is the only
> > layer
> > > > of
> > > > > > > > > > indirection
> > > > > > > > > > > at the API entry-point, but then once the set of
> > > > > > > activePrincipalRoles
> > > > > > > > > is
> > > > > > > > > > > resolved, the underlying authz engine doesn't behave
> any
> > > > > > > differently
> > > > > > > > > from
> > > > > > > > > > > normal Polaris authz.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <
> > > > flyrain000@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks everyone for the discussion!
> > > > > > > > > > > >
> > > > > > > > > > > > As a next step discussed in today's community sync,
> we
> > > will
> > > > > > start
> > > > > > > > > > reviewing
> > > > > > > > > > > > the spec change.
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Dennis, is this PR ready for review?
> > > > > > > > > > > > https://github.com/apache/polaris/pull/5446
> > > > > > > > > > > >
> > > > > > > > > > > > Yufei
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <
> > > > > > > > > travismbowen@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for this discussion Dennis!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Wanted to weigh in with my thoughts and questions.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Do you prefer /api/shares-management/v1/
> > > > > > > > > > > > > > instead of adding to /api/management/v1?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > While I can see the argument for a share specific
> > > > > management
> > > > > > > > > endpoint
> > > > > > > > > > > > there
> > > > > > > > > > > > > doesn't seem to be a clear obvious choice from
> what I
> > > can
> > > > > see
> > > > > > > - I
> > > > > > > > > > lean
> > > > > > > > > > > > > towards /api/management as being the central place
> > for
> > > > all
> > > > > > the
> > > > > > > > > > > > management -
> > > > > > > > > > > > > share or not. As long as we're careful not to leak
> > > share
> > > > > > types
> > > > > > > or
> > > > > > > > > non
> > > > > > > > > > > > share
> > > > > > > > > > > > > types into the wrong api (which could be helped by
> > > having
> > > > > > > > separate
> > > > > > > > > > yamls)
> > > > > > > > > > > > > then it feels reasonable to have all under the
> > > management
> > > > > > api.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Also went through the doc and PR and made several
> > > > comments
> > > > > as
> > > > > > > > well.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri
> Bourlatchkov <
> > > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Prithvi mentioned ExternalConsumer and
> principals.
> > > I'd
> > > > > like
> > > > > > > to
> > > > > > > > > > > > > > reinforce the need to further discuss "consumers"
> > and
> > > > > > > > principals
> > > > > > > > > > and
> > > > > > > > > > > > how
> > > > > > > > > > > > > > they relate to IdP and Polaris Authentication.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > From my POV, share access control should not be a
> > new
> > > > > > > mechanism
> > > > > > > > > but
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > build on top of the existing extensible
> > > authentication
> > > > /
> > > > > > > > > > authorization
> > > > > > > > > > > > > > framework. Supporting external IdP for shares
> would
> > > be
> > > > a
> > > > > > > > valuable
> > > > > > > > > > > > feature
> > > > > > > > > > > > > > IMHO.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Side note: I'm posting comments on PR 5446 too.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <
> > > > > > > > > > prithvisivasankar@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you for reviving this, and for the
> concrete
> > > > spec
> > > > > in
> > > > > > > PR
> > > > > > > > > > 5446..
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > journeys doc was especially helpful for seeing
> > the
> > > > > > > end-to-end
> > > > > > > > > > flow.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I agree this is worth doing, and that v1 is
> more
> > > > than a
> > > > > > > > > tutorial
> > > > > > > > > > on
> > > > > > > > > > > > > > today's
> > > > > > > > > > > > > > > RBAC. A partner principal on /api/catalog is
> > > already
> > > > > > > > possible;
> > > > > > > > > > what
> > > > > > > > > > > > > seems
> > > > > > > > > > > > > > > valuable is a first-class share (enumerated
> > > members,
> > > > a
> > > > > > > > > restricted
> > > > > > > > > > > > > > consumer,
> > > > > > > > > > > > > > > a listing as the binding and audit unit) plus a
> > > > > > segregated,
> > > > > > > > > > read-only
> > > > > > > > > > > > > > > Iceberg REST surface with credential vending.
> > Stock
> > > > > > Iceberg
> > > > > > > > > REST
> > > > > > > > > > as
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > consumer protocol, and keeping Flight and
> > > > > > > Polaris-to-Polaris
> > > > > > > > > > > > federation
> > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > of v1, all sound right to me.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I don't have a strong view yet on the URL
> layout
> > > > > > > (management
> > > > > > > > vs
> > > > > > > > > > > > shares
> > > > > > > > > > > > > > > prefix, nested vs top-level, prefix keyed by
> > share
> > > vs
> > > > > > > > listing)
> > > > > > > > > > and am
> > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > to follow whatever the community prefers there.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > A few things I wanted to check on the spec
> > itself,
> > > > > > several
> > > > > > > of
> > > > > > > > > > these
> > > > > > > > > > > > > echo
> > > > > > > > > > > > > > > points already raised on the list:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. Membership as source of truth. If a hidden
> > > catalog
> > > > > > role
> > > > > > > is
> > > > > > > > > > > > > > independently
> > > > > > > > > > > > > > > grantable, a share could be widened without
> going
> > > > > through
> > > > > > > the
> > > > > > > > > > share
> > > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > > Would it make sense to say the data plane
> > > authorizes
> > > > > > > against
> > > > > > > > > > > > > membership +
> > > > > > > > > > > > > > > listing, and that any backing roles are not
> > > > > user-visible
> > > > > > /
> > > > > > > > not
> > > > > > > > > > > > > > > independently grantable?
> > > > > > > > > > > > > > > 2. ExternalConsumer and principals. The Aug 7
> > recap
> > > > > > already
> > > > > > > > > > noted we
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > not assume a PrincipalEntity behind every
> > > > > > ExternalConsumer.
> > > > > > > > The
> > > > > > > > > > > > > journeys
> > > > > > > > > > > > > > > still describe granting the consumer a share
> > > > principal
> > > > > > > role,
> > > > > > > > I
> > > > > > > > > > assume
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > is an implementation sketch for v1
> > > client_id/secret,
> > > > > not
> > > > > > > the
> > > > > > > > > > public
> > > > > > > > > > > > > > model?
> > > > > > > > > > > > > > > 3. Listing expiration. JB's original note was
> > > "share
> > > > > > these
> > > > > > > > > tables
> > > > > > > > > > > > with
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > consumer until this date." Credential expiry
> is a
> > > bit
> > > > > > > > > different.
> > > > > > > > > > > > Would
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > listing-level expiration (and maybe
> > > ACTIVE/SUSPENDED)
> > > > > be
> > > > > > in
> > > > > > > > > > scope for
> > > > > > > > > > > > > v1?
> > > > > > > > > > > > > > > 4. Time-travel. As raised in the community
> sync,
> > > > > sharing
> > > > > > > via
> > > > > > > > > > Iceberg
> > > > > > > > > > > > > REST
> > > > > > > > > > > > > > > means historical snapshots remain readable.
> > Totally
> > > > > fine
> > > > > > > for
> > > > > > > > v1
> > > > > > > > > > to
> > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > only full history, would an explicit field on
> the
> > > > share
> > > > > > > help
> > > > > > > > > so a
> > > > > > > > > > > > later
> > > > > > > > > > > > > > > CURRENT_ONLY option is not a breaking surprise?
> > > > > > > > > > > > > > > 5. Consumer data plane. The current file is
> > > > > control-plane
> > > > > > > > only,
> > > > > > > > > > which
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > good first step. It might help to list the
> > Iceberg
> > > > > routes
> > > > > > > > that
> > > > > > > > > > are
> > > > > > > > > > > > > > mounted,
> > > > > > > > > > > > > > > that writes 404, that non-members 404, and that
> > > > /config
> > > > > > > does
> > > > > > > > > not
> > > > > > > > > > > > return
> > > > > > > > > > > > > > > provider catalog properties, so implementations
> > > don't
> > > > > > > > diverge.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Two smaller notes, if useful: in the J1 journey
> > the
> > > > > > > consumer
> > > > > > > > > > still
> > > > > > > > > > > > > posts
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > /api/catalog/v1/oauth/tokens, if partners are
> not
> > > > > > supposed
> > > > > > > to
> > > > > > > > > > touch
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > internal catalog API, should the default token
> > > > endpoint
> > > > > > > live
> > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > > > > > shares
> > > > > > > > > > > > > > > surface (or an external IdP)? And on Dmitri's
> > point
> > > > > about
> > > > > > > > share
> > > > > > > > > > > > > operators
> > > > > > > > > > > > > > > vs catalog operators, a SHARE_MANAGE privilege
> on
> > > the
> > > > > > bound
> > > > > > > > > > catalog
> > > > > > > > > > > > > > > (independent of CATALOG_MANAGE_ACCESS) would
> make
> > > > that
> > > > > > work
> > > > > > > > on
> > > > > > > > > > any
> > > > > > > > > > > > URL
> > > > > > > > > > > > > > > prefix.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Happy to review the next revision, and thanks
> > again
> > > > to
> > > > > > both
> > > > > > > > of
> > > > > > > > > > you
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > driving this.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > > Prithvi S
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <
> > > > > > dhuo@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > As discussed live, looks like we both might
> > have
> > > > some
> > > > > > > > drafts
> > > > > > > > > > we can
> > > > > > > > > > > > > > > compare
> > > > > > > > > > > > > > > > and choose/merge from :)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Here's my current draft spec PR; it opts for
> my
> > > > > > preferred
> > > > > > > > > > syntax
> > > > > > > > > > > > > > choices
> > > > > > > > > > > > > > > > for now (/api/management/ for
> share-management,
> > > > > > > > /api/shares/
> > > > > > > > > > for
> > > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > data-plane):
> > > > > > https://github.com/apache/polaris/pull/5446
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Since we were talking about preferring
> mdfiles
> > > for
> > > > > > design
> > > > > > > > > > iteration
> > > > > > > > > > > > > > > here's
> > > > > > > > > > > > > > > > a companion design-addendum which just
> > enumerates
> > > > > what
> > > > > > > the
> > > > > > > > > > > > end-to-end
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > journeys look like syntactically for each
> > > > combination
> > > > > > of
> > > > > > > > > design
> > > > > > > > > > > > > > choices:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Please let me know if anyone prefers a Google
> > Doc
> > > > > > version
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > > > design-addendum for easier commenting, or if
> > the
> > > > > > > > > > design-addendum
> > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > should also be modeled as a PR on some
> > > > free-floating
> > > > > > > branch
> > > > > > > > > > just to
> > > > > > > > > > > > > > > capture
> > > > > > > > > > > > > > > > comments.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste
> > > > Onofré <
> > > > > > > > > > > > jb@nanthrax.net
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I am resuming work on this proposal.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Regarding our recent discussion, I will
> move
> > > > > forward
> > > > > > > with
> > > > > > > > > > > > opening a
> > > > > > > > > > > > > > > > > draft PR for the /api/shares/v1 endpoint to
> > > help
> > > > > > > > illustrate
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > design.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I will update this thread once the PR is
> > ready.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > JB
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri
> > > > Bourlatchkov
> > > > > <
> > > > > > > > > > > > > dimas@apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Dennis,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the recap!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I was thinking that share management is
> > > > probably
> > > > > > > > distinct
> > > > > > > > > > from
> > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > management. A user who already has a
> > > > > realm+catalog
> > > > > > > may
> > > > > > > > > > have an
> > > > > > > > > > > > > > option
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > manage shares, but not the catalog
> itself.
> > > > Would
> > > > > it
> > > > > > > > make
> > > > > > > > > > sense
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > > the share management API from the general
> > > > Polaris
> > > > > > > > > > management
> > > > > > > > > > > > > API? I
> > > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > using a different prefix for URIs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For example: /api/shares/v1/
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Obviously clients should not make any
> > > > assumptions
> > > > > > > about
> > > > > > > > > the
> > > > > > > > > > > > base
> > > > > > > > > > > > > > path
> > > > > > > > > > > > > > > > > > (/api/shares), which could be different
> in
> > > > > > different
> > > > > > > > > > > > deployments
> > > > > > > > > > > > > > > (e.g.
> > > > > > > > > > > > > > > > > > going through a gateway). Clients must be
> > > able
> > > > to
> > > > > > > take
> > > > > > > > > the
> > > > > > > > > > base
> > > > > > > > > > > > > URI
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > config option. Path segments after /v1/
> > will
> > > be
> > > > > > > > specified
> > > > > > > > > > by
> > > > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > > (Open
> > > > > > > > > > > > > > > > > > API yaml).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis
> Huo <
> > > > > > > > > dhuo@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Just to recap some of the discussion
> from
> > > > > today's
> > > > > > > > > > community
> > > > > > > > > > > > > sync,
> > > > > > > > > > > > > > > > > Dmitri,
> > > > > > > > > > > > > > > > > > > Robert, and Alex helped raise a few
> > > > > > considerations
> > > > > > > to
> > > > > > > > > > roll
> > > > > > > > > > > > into
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > next
> > > > > > > > > > > > > > > > > > > design/prototyping phase:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. Given the way sharing is packaged
> as a
> > > > > > separate
> > > > > > > > > > thing, it
> > > > > > > > > > > > > may
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > obvious to users that historical
> > snapshots
> > > > are
> > > > > > > > equally
> > > > > > > > > > > > > available
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > consumers as the latest snapshot (by
> > virtue
> > > > of
> > > > > > > > sharing
> > > > > > > > > > via
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > standard IRC
> > > > > > > > > > > > > > > > > > > protocol) - we should make sure to
> > document
> > > > > that
> > > > > > > fact
> > > > > > > > > > well,
> > > > > > > > > > > > > > > including
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > pitfall that deleting rows by updating
> a
> > > > table
> > > > > > > > doesn't
> > > > > > > > > > > > > > necessarily
> > > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > > those rows become inaccessible to
> > consumers
> > > > > > > > > > > > > > > > > > > 2. Make sure not to hard-code
> assumptions
> > > > about
> > > > > > the
> > > > > > > > > > existence
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > internal PrincipalEntity backing an
> > > > > > > ExternalConsumer
> > > > > > > > -
> > > > > > > > > > even
> > > > > > > > > > > > if
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > > > MVP implementation uses a nested/hidden
> > > > > Principal
> > > > > > > of
> > > > > > > > > some
> > > > > > > > > > > > type
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > holds a
> > > > > > > > > > > > > > > > > > > Polaris-managed
> client_id/client_secret,
> > > the
> > > > > > > > 3rd-party
> > > > > > > > > > IdP
> > > > > > > > > > > > use
> > > > > > > > > > > > > > > cases
> > > > > > > > > > > > > > > > > > > require "implicit" Principals resolved
> > via
> > > > the
> > > > > > > oauth
> > > > > > > > > > token.
> > > > > > > > > > > > In
> > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > cases
> > > > > > > > > > > > > > > > > > > the ExternalConsumer contains the
> > metadata
> > > > > > > > coordinating
> > > > > > > > > > the
> > > > > > > > > > > > > > > three-way
> > > > > > > > > > > > > > > > > > > handshake between the data-provider,
> > > > > > data-consumer,
> > > > > > > > and
> > > > > > > > > > IdP,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > no PrincipalEntity underlying it.
> > > > > > > > > > > > > > > > > > > 3. Longer-term, we may want to tackle
> > > > > > > "on-behalf-of"
> > > > > > > > > > > > semantics
> > > > > > > > > > > > > -
> > > > > > > > > > > > > > > > > initially,
> > > > > > > > > > > > > > > > > > > the way an ExternalConsumer works is
> that
> > > the
> > > > > > > > > > finer-grained
> > > > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > share a single ExternalConsumer
> identity,
> > > > just
> > > > > > like
> > > > > > > > > how a
> > > > > > > > > > > > > > FEDERATED
> > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > shares a single service principal in
> its
> > > > > > > > > > ConnectionConfig.
> > > > > > > > > > > > But
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > cases the consumer may want some
> > end-to-end
> > > > > > > > attribution
> > > > > > > > > > of
> > > > > > > > > > > > > > > > fine-grained
> > > > > > > > > > > > > > > > > > > users, like "on-behalf-of: bob @
> > > > > > > consumercompany.com
> > > > > > > > "
> > > > > > > > > > when
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > ExternalConsumer implicit principal is
> > > > > something
> > > > > > > like
> > > > > > > > > > > > > > > > > > > "externalconsumer$consumercompany"
> > > > > > > > > > > > > > > > > > > 4. We can iterate more rapidly on a
> draft
> > > > spec
> > > > > PR
> > > > > > > and
> > > > > > > > > > also
> > > > > > > > > > > > > expand
> > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > detailed technical design
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis
> > Huo
> > > <
> > > > > > > > > > dhuo@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks JB for reviving this thread,
> and
> > > > > belated
> > > > > > > > > thanks
> > > > > > > > > > to
> > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > reviewers
> > > > > > > > > > > > > > > > > > > > who have left feedback in the doc or
> > this
> > > > > > thread
> > > > > > > so
> > > > > > > > > > far!
> > > > > > > > > > > > And
> > > > > > > > > > > > > > > > > apologies
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > the silence on this, it's still been
> on
> > > my
> > > > > > radar
> > > > > > > > but
> > > > > > > > > I
> > > > > > > > > > got
> > > > > > > > > > > > > > > > > distracted by
> > > > > > > > > > > > > > > > > > > > some other work for awhile.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Echoing JB's summary, the proposal is
> > > > indeed
> > > > > > that
> > > > > > > > > we're
> > > > > > > > > > > > > mostly
> > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > > reusing existing Authz/RBAC
> machinery,
> > > and
> > > > > the
> > > > > > > > > > translation
> > > > > > > > > > > > > > layer
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > sharing concepts sits *on top of* the
> > > > > existing
> > > > > > > > authz
> > > > > > > > > > engine
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > much
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > So even if we're adding layers of
> > > > indirection
> > > > > > on
> > > > > > > > > > > > > > > > > ExternalConsumer/Listing
> > > > > > > > > > > > > > > > > > > > ->
> > > > > Principal/PrincipalRole/CatalogRole+grants,
> > > > > > > the
> > > > > > > > > > parts
> > > > > > > > > > > > that
> > > > > > > > > > > > > > end
> > > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > > the underlying authz just reuse
> > existing
> > > > > > > Authorizer
> > > > > > > > > > logic.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Based on some other offline
> discussion
> > > with
> > > > > > > > Dmitri, I
> > > > > > > > > > > > > realized
> > > > > > > > > > > > > > > why
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > discussion around "requirements" may
> > have
> > > > > been
> > > > > > > > > > confusing:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. My attempts to shorten the doc
> > > > > inadvertently
> > > > > > > cut
> > > > > > > > > out
> > > > > > > > > > > > some
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > "abstract requirements" formulations
> > > > > > > > > > > > > > > > > > > > 2. The project in itself is somewhat
> > more
> > > > of
> > > > > a
> > > > > > > > "form
> > > > > > > > > > *as*
> > > > > > > > > > > > > > > function"
> > > > > > > > > > > > > > > > > > > > feature than *strictly* functional in
> > > > nature
> > > > > --
> > > > > > > for
> > > > > > > > > > > > example,
> > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > correctly
> > > > > > > > > > > > > > > > > > > > pointed out that "sharing" via the
> > > existing
> > > > > > > > > > RBAC/Principals
> > > > > > > > > > > > > > > syntax
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > already *possible* just by letting a
> > > > > cross-org
> > > > > > > > > > "consumer"
> > > > > > > > > > > > be
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > > > > > principal and access your own
> company's
> > > > main
> > > > > > > > Polaris
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > endpoints.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > For (1) I added this to the
> > Requirements
> > > > > > section
> > > > > > > in
> > > > > > > > > > the doc
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > better
> > > > > > > > > > > > > > > > > > > > clarify the abstract form of the data
> > > model
> > > > > > > > element:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > *A. Sharing Container*: Container
> > concept
> > > > for
> > > > > > > > holding
> > > > > > > > > > > > things
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > want to
> > > > > > > > > > > > > > > > > > > > share - “ShareEntity”
> > > > > > > > > > > > > > > > > > > > *B. Consumer Identity*: Special
> flavor
> > of
> > > > > > > > > > principal-like
> > > > > > > > > > > > > entity
> > > > > > > > > > > > > > > > > > > > representing a consumer relationship
> -
> > > > > > > > > > “ExternalConsumer”
> > > > > > > > > > > > > > > > > > > > *C. Map Containers to Consumers*: A
> way
> > > to
> > > > > > > specify
> > > > > > > > > > letting
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > > consumer access a given sharing
> > > container -
> > > > > > > > “Listing”
> > > > > > > > > > > > > > > > > > > > *D. Consumer Endpoint*: A segregated
> > API
> > > > > > > > access-point
> > > > > > > > > > for
> > > > > > > > > > > > > > > > consumers -
> > > > > > > > > > > > > > > > > > > > “EndpointConfig”
> > > > > > > > > > > > > > > > > > > > *E. Objects in Containers*: A way to
> > > > specify
> > > > > > > which
> > > > > > > > > > > > in-catalog
> > > > > > > > > > > > > > > > objects
> > > > > > > > > > > > > > > > > > > > belong in the big sharing container -
> > > > > > > > > “ShareMembership”
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This still bakes in a bit of
> > > implementation
> > > > > > > detail
> > > > > > > > > > since it
> > > > > > > > > > > > > > begs
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > question why not just let core
> > > RBAC/grants
> > > > > > handle
> > > > > > > > (C)
> > > > > > > > > > and
> > > > > > > > > > > > > (E).
> > > > > > > > > > > > > > To
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > end,
> > > > > > > > > > > > > > > > > > > > and addressing problem (2) of where
> the
> > > > "form
> > > > > > > *is*
> > > > > > > > > > > > > function", I
> > > > > > > > > > > > > > > > > added a
> > > > > > > > > > > > > > > > > > > > "Concept Mapping" tab:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think the details of more advanced
> > > > features
> > > > > > can
> > > > > > > > > > still be
> > > > > > > > > > > > > > hashed
> > > > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > > > > incrementally without blocking work
> on
> > an
> > > > MVP
> > > > > > > (and
> > > > > > > > > > > > certainly
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > MVP
> > > > > > > > > > > > > > > > > > > scope
> > > > > > > > > > > > > > > > > > > > could be trimmed so we can
> > > > > launch-and-iterate),
> > > > > > > so
> > > > > > > > as
> > > > > > > > > > long
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > > there's
> > > > > > > > > > > > > > > > > > > > community alignment on the general
> > > > direction
> > > > > I
> > > > > > > > > propose
> > > > > > > > > > we
> > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > move
> > > > > > > > > > > > > > > > > > > > forward on basic building blocks. Or
> if
> > > > > > building
> > > > > > > > > blocks
> > > > > > > > > > > > > > > themselves
> > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > controversial we could still proceed
> > with
> > > > > some
> > > > > > > > > > prototyping
> > > > > > > > > > > > > > across
> > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > options.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM
> > > > > Jean-Baptiste
> > > > > > > > > Onofré <
> > > > > > > > > > > > > > > > > jb@nanthrax.net>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >> Hi everyone,
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> I would like to resume the
> discussion
> > > > about
> > > > > > Data
> > > > > > > > > > Sharing
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > > > > >> and propose focusing on three major
> > > items
> > > > to
> > > > > > > help
> > > > > > > > us
> > > > > > > > > > move
> > > > > > > > > > > > > > > forward.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> 1. What is data sharing?
> > > > > > > > > > > > > > > > > > > >> In a nutshell, it is an open
> > > protocol
> > > > > > > proposal
> > > > > > > > > > > > allowing
> > > > > > > > > > > > > > > > > > > >> organizations to share live data
> > without
> > > > > > copying
> > > > > > > > or
> > > > > > > > > > > > > > replication.
> > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > >> main goal is zero-copy sharing
> > achieved
> > > > > > through
> > > > > > > > > shared
> > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > >> authorization.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> On Polaris, this would mean:
> > > > > > > > > > > > > > > > > > > >> a. The consumer is authenticated as
> > > usual.
> > > > > > > > > > > > > > > > > > > >> b. The Polaris server provides the
> > > > consumer
> > > > > > with
> > > > > > > > > > direct
> > > > > > > > > > > > > access
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >> data files (e.g., Parquet, as
> > specified
> > > in
> > > > > the
> > > > > > > > > > > > > metadata.json).
> > > > > > > > > > > > > > > > > > > >> c. The consumer reads the data files
> > > > > directly
> > > > > > > > using
> > > > > > > > > > their
> > > > > > > > > > > > > > > > preferred
> > > > > > > > > > > > > > > > > > > >> tool (e.g., a query engine).
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> While Polaris already has most of
> the
> > > > > > necessary
> > > > > > > > > > underlying
> > > > > > > > > > > > > > > > > > > >> capabilities, we need clear
> > > > > branding/packaging
> > > > > > > > > around
> > > > > > > > > > this
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > case.
> > > > > > > > > > > > > > > > > > > >> Specifically, this means:
> > > > > > > > > > > > > > > > > > > >> a. A share is a dedicated catalog in
> > the
> > > > > > Polaris
> > > > > > > > > > server,
> > > > > > > > > > > > or
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > >> role scoped to specific namespaces.
> > > > > > > > > > > > > > > > > > > >> b. A recipient is represented by a
> > > > principal
> > > > > > > and a
> > > > > > > > > > > > principal
> > > > > > > > > > > > > > > role.
> > > > > > > > > > > > > > > > > > > >> c. A consumer profile consists of
> the
> > > > > catalog
> > > > > > > URI
> > > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > > > OAuth2
> > > > > > > > > > > > > > > > > > > >> client_id/client_secret.
> > > > > > > > > > > > > > > > > > > >> d. We use credential vending
> > (temporary
> > > > STS
> > > > > > > > > > credentials)
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > > > > >> access to the data files.
> > > > > > > > > > > > > > > > > > > >> e. We use the Polaris IRC layer to
> > > expose
> > > > > the
> > > > > > > > > > metadata and
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > > > >> locations/access.
> > > > > > > > > > > > > > > > > > > >> f. We can also support
> > > Polaris-to-Polaris
> > > > > > > sharing
> > > > > > > > > via
> > > > > > > > > > > > > > > Federation.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Concretely:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> - On the provider side, we use the
> > > > > standard
> > > > > > > > > Polaris
> > > > > > > > > > RBAC
> > > > > > > > > > > > > > model
> > > > > > > > > > > > > > > > > > > >> (assigning grants to catalog roles,
> > and
> > > > > > > assigning
> > > > > > > > > > those
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > roles
> > > > > > > > > > > > > > > > > > > >> to principal roles):
> > > > > > > > > > > > > > > > > > > >> - Create a dedicated principal
> > > with
> > > > > its
> > > > > > > own
> > > > > > > > > > > > > > > > > > > client_id/client_secret.
> > > > > > > > > > > > > > > > > > > >> - Create a catalog role with
> > > > > > > > TABLE_READ_DATA,
> > > > > > > > > > > > > > TABLE_LIST,
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > >> NAMESPACE_LIST grants on the
> > namespaces
> > > to
> > > > > be
> > > > > > > > shared
> > > > > > > > > > (no
> > > > > > > > > > > > > write
> > > > > > > > > > > > > > > > > > > >> grants).
> > > > > > > > > > > > > > > > > > > >> - Create a principal role per
> > > > > > destination,
> > > > > > > > > > attach
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > >> role, and assign the principal.
> > > > > > > > > > > > > > > > > > > >> - The destination engine uses
> > > > > loadTable
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >> "X-Iceberg-Access-Delegation:
> > > > > > > vended-credentials"
> > > > > > > > > > header.
> > > > > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > > > >> emits temporary, read-only storage
> > > > > > credentials,
> > > > > > > > > > allowing
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > engine
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > >> read the data files directly from
> the
> > > > > owner's
> > > > > > > > bucket
> > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > accessing
> > > > > > > > > > > > > > > > > > > >> the rest of the bucket. This
> achieves
> > > > > > zero-copy
> > > > > > > > > > sharing.
> > > > > > > > > > > > > > > > > > > >> - On the recipient/consumer side,
> no
> > > > > special
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > >> needed; any IRC client can use it.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> 2. What is data sharing not?
> > > > > > > > > > > > > > > > > > > >> I previously considered adding
> an
> > > > Arrow
> > > > > > > Flight
> > > > > > > > > > > > endpoint
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > >> Polaris, but I no longer think it
> is a
> > > > good
> > > > > > idea
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > > initial
> > > > > > > > > > > > > > > > > > > >> phase. An Arrow Flight endpoint
> would
> > > > serve
> > > > > > the
> > > > > > > > data
> > > > > > > > > > > > > directly,
> > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > >> would require Polaris to read the
> data
> > > and
> > > > > > > expose
> > > > > > > > > it.
> > > > > > > > > > I am
> > > > > > > > > > > > > > > against
> > > > > > > > > > > > > > > > > > > >> embedding a query engine in Polaris,
> > as
> > > > this
> > > > > > > tight
> > > > > > > > > > > > coupling
> > > > > > > > > > > > > > > feels
> > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > >> an anti-pattern. While we could use
> a
> > > > > > delegation
> > > > > > > > > > service
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >> Flight endpoint, I think it should
> be
> > > > > > considered
> > > > > > > > > for a
> > > > > > > > > > > > later
> > > > > > > > > > > > > > > > phase,
> > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > >> at all, since a Flight endpoint
> might
> > be
> > > > > > better
> > > > > > > > > > suited as
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > > > > >> responsibility and could encourage
> > data
> > > > > > copying.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> The first phase should focus on
> > > packaging
> > > > > and
> > > > > > > > > > branding the
> > > > > > > > > > > > > > > > > > > >> capabilities we already have.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> 3. What could the Data Sharing
> > > "package"
> > > > > look
> > > > > > > > like
> > > > > > > > > in
> > > > > > > > > > > > > > Polaris?
> > > > > > > > > > > > > > > > > > > >> To make Data Sharing obvious and
> > > easy
> > > > to
> > > > > > > use,
> > > > > > > > we
> > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > > > > >> the missing concept of a "share"—a
> > named
> > > > > > > > definition
> > > > > > > > > > that
> > > > > > > > > > > > > users
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > >> list, discover, and define. The goal
> > is
> > > to
> > > > > > > > > > materialize the
> > > > > > > > > > > > > > > user's
> > > > > > > > > > > > > > > > > > > >> intent ("I want to share these
> tables
> > > with
> > > > > > this
> > > > > > > > > > consumer
> > > > > > > > > > > > > until
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > >> date") into standard Polaris
> entities
> > > > > > > (principal,
> > > > > > > > > > > > principal
> > > > > > > > > > > > > > > role,
> > > > > > > > > > > > > > > > > > > >> catalog role, and grants).
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Therefore, I propose adding a
> /shares
> > > > > endpoint
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > > > > >> Management API to package and
> > > materialize
> > > > > > these
> > > > > > > > > share
> > > > > > > > > > > > > > > definitions
> > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > >> Polaris entities. This would
> provide a
> > > > > usable,
> > > > > > > > > > > > first-version
> > > > > > > > > > > > > > > > package
> > > > > > > > > > > > > > > > > > > >> for Polaris Data Sharing.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> I hope this addresses Dmitri's
> > questions
> > > > and
> > > > > > > > > provides
> > > > > > > > > > a
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > concrete
> > > > > > > > > > > > > > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Thoughts?
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Regards,
> > > > > > > > > > > > > > > > > > > >> JB
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM
> Dennis
> > > > Huo <
> > > > > > > > > > > > dhuo@apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >> > Hi All,
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >> > One big emerging enterprise use
> case
> > > > > coming
> > > > > > up
> > > > > > > > as
> > > > > > > > > > more
> > > > > > > > > > > > > > people
> > > > > > > > > > > > > > > > > > > >> consolidate
> > > > > > > > > > > > > > > > > > > >> > Data Lakehouses and Catalogs is
> > > > something
> > > > > > > > commonly
> > > > > > > > > > known
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > "Data
> > > > > > > > > > > > > > > > > > > >> Sharing",
> > > > > > > > > > > > > > > > > > > >> > an more specifically over the
> course
> > > of
> > > > > > > adoption
> > > > > > > > > of
> > > > > > > > > > open
> > > > > > > > > > > > > > table
> > > > > > > > > > > > > > > > > formats
> > > > > > > > > > > > > > > > > > > >> > "Open Data Sharing".
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >> > Examples of existing managed
> service
> > > > > > > providers'
> > > > > > > > > Data
> > > > > > > > > > > > > Sharing
> > > > > > > > > > > > > > > > > features:
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > https://www.databricks.com/product/delta-sharing
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > >
> > > > > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >> > The basic idea is that when you
> > share
> > > > data
> > > > > > > > between
> > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > companies,
> > > > > > > > > > > > > > > > > > > >> you
> > > > > > > > > > > > > > > > > > > >> > need a first-class
> > > governance/management
> > > > > > layer
> > > > > > > > and
> > > > > > > > > > extra
> > > > > > > > > > > > > > > > > > > >> bells-and-whistles
> > > > > > > > > > > > > > > > > > > >> > that are distinct from just the
> > basic
> > > > > > > > capabilities
> > > > > > > > > > of
> > > > > > > > > > > > RBAC
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > >> generalized
> > > > > > > > > > > > > > > > > > > >> > access-control (i.e. if you're
> > sharing
> > > > > > across
> > > > > > > > > > > > > > > > partially-untrusted
> > > > > > > > > > > > > > > > > org
> > > > > > > > > > > > > > > > > > > >> > boundaries, you don't just let the
> > > > > consumer
> > > > > > > > > > organization
> > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > > > >> > datalake like one of your own
> > > > employees).
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >> > JB and I put together this
> > high-level
> > > > > > proposal
> > > > > > > > for
> > > > > > > > > > > > > > supporting
> > > > > > > > > > > > > > > > Open
> > > > > > > > > > > > > > > > > > > >> Sharing
> > > > > > > > > > > > > > > > > > > >> > in Polaris:
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >> > Tentatively, it means adding ~5
> > > logical
> > > > > data
> > > > > > > > model
> > > > > > > > > > > > > > constructs,
> > > > > > > > > > > > > > > > > some of
> > > > > > > > > > > > > > > > > > > >> > which may be a first-class
> > > PolarisEntity
> > > > > > type,
> > > > > > > > > > others
> > > > > > > > > > > > > > subtypes
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > >> existing
> > > > > > > > > > > > > > > > > > > >> > entities, and others just a nested
> > > > > > construct:
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >> > - ShareEntity (would behave
> > > similarly
> > > > > to
> > > > > > a
> > > > > > > > > > Catalog)
> > > > > > > > > > > > > > > > > > > >> > - ExternalConsumer (mostly
> > inherits
> > > > > from
> > > > > > > > > > Principal)
> > > > > > > > > > > > > > > > > > > >> > - Listing (Similar to a "role
> > > grant"
> > > > > but
> > > > > > > has
> > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > metadata)
> > > > > > > > > > > > > > > > > > > >> > - EndpointConfig (nested config
> > > under
> > > > > > > > Listing)
> > > > > > > > > > > > > > > > > > > >> > - ShareMembership (Similar to a
> > > > > > "securable
> > > > > > > > > > grant" but
> > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > >> metadata)
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >> > Feedback/comments welcome! I'll
> also
> > > > bring
> > > > > > it
> > > > > > > up
> > > > > > > > > for
> > > > > > > > > > > > live
> > > > > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > >> > there's time in the community
> sync.
> > > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > > >> > Cheers,
> > > > > > > > > > > > > > > > > > > >> > Dennis
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Dennis,
Thanks for the detailed path forward.
The stages / compartmentalized approach make sense to me.
1) The Shares API defines "what to share". It does not depend on anything
else.
2) The Authorizer SPI changes provide a way to express Share access
requests.
3) Internal RBAC implements (2) on top of the internal Principals and Roles.
I suppose ExternalConsumer will fit here.
4) External Authorizer implementations adapt to (2) and allow users to
configure access rules in their respective configuration styles.
Does that make sense?
Thanks,
Dmitri.
On Thu, Oct 22, 2026 at 6:58 PM Dennis Huo <dh...@apache.org> wrote:
> Right, sorry, to clarify, my stated bullets in the last email were my
> hypothetical portrayal of what your proposed core set of APIs would be, not
> what my current proposal says (so that you can confirm or deny whether
> those three bullets convey your ideal state).
>
> It sounds like (2) and (3) match your intent, and (1) is exactly where the
> current spec PR differs, since it contains consumers and listings.
>
> For the particular use cases that went into the scoping for the Polaris
> Shares proposal, the ExternalConsumer is fairly fundamental to extracting
> as a first-class object.
>
> But you're right that there could be a trimmed scoping that allows
> implementors to fully abstract away the ExternalConsumer/Listing semantics
> instead. In the same way, there's even a fully-trimmed formulation where we
> don't implement *any* management APIs at all: Where "Sharing" is just data
> providers providing dedicated lightweight Polaris servers, or adding a very
> simple config-file-based mapping for what data to make accessible to
> dedicated "sharing" Principals or PrincipalRoles.
>
> We could reconcile this by also moving the ExternalConsumer/Listings APIs
> to the same "optional" yaml file I moved the other
> ConsumerCredential-related endpoints into and keeping the "core Shares API"
> limited to what you wanted. Broadly, the two pillars of the proposal are:
>
> 1. *What* to share
> 2. *With whom* to share
>
> The core Shares API would only cover the first, in line with your proposal:
> CRUD on Shares and the securables contained in them (listShares,
> createShare, getShare, updateShare, deleteShare, listShareMembers,
> updateShareMembers).
>
> The optional file would cover the second: ExternalConsumers, Listings, the
> endpoint configuration, and credentials for the polaris-standalone case.
>
> Polaris service-runners could then decide if they just want sharing
> consumers to live in the exact same identity domain as their other native
> Polaris readers, or segregate them by enabling the "With whom" half of the
> API.
>
> One thing to discuss for the "core-only" case - today the management API
> doesn't syntactically support granting a share to a PrincipalRole, so we'd
> need to extend grant types or expose some of the internals of a share's
> backing catalog role. I guess we can iteratively refine any necessary API
> changes as we discover these in the core-only implementation.
>
> Note that the "with whom" half is necessary for the full "segregation"
> reqiurements stated in the original proposal doc, such as preventing
> assignment to other roles or accessing the "normal" catalog endpoints,
> which is why it can't be *fully* subsumed by only the native Polaris
> management API.
>
> It also doesn't conflict with external IdPs: Compare to Databricks'
> RECIPIENT for example, which can wrap a Databricks identity, a
> provider-issued token, or the recipients own IdP through an OIDC federatoin
> policy, and shares are granted uniformly to it in all three cases:
> https://docs.databricks.com/aws/en/delta-sharing/create-recipient-oidc-fed
>
> Even in the full API model my PR proposes, we could let the Listing be
> something the authorizer looks at alongside other authz conext, so the
> pluggable authorizers/OPA/Ranger/etc can integrate as the authoritative
> layer.
>
>
> On Thu, Oct 22, 2026 at 2:00 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Dennis,
> >
> > > - Shares API defines only shares and the objects placed in shares, with
> > no
> > > Consumer or Listing
> >
> > POST /shares/{shareName}/listings taking a CreateListingRequest with a
> > `consumerName` is still part of `spec/polaris-shares-api.yaml`.
> >
> > I'm kind of confused about the scope of "shares" API vs. other APIs :)
> > Where are the boundaries?
> >
> > Would could be the minimal isoalated API sufficient for managing Shares?
> >
> > > - Controlling who can read shares is only ordinary Polaris authz, e.g.
> > > assignPrincipalRoletoPrincipal or
> > addGrantOnSecurableToPrincipalRole(share,
> > > principalRole)
> >
> > Sounds good.
> >
> > > - The partner organization consuming the share is just a principal,
> which
> > > could either be Polaris-managed or from an external IdP
> >
> > This also sounds good to me.
> >
> > I guess the only concern / confusion area is around Listing / Consumer
> > (first point above).
> >
> > Cheers,
> > Dmitri.
> >
> >
> > On Thu, Oct 22, 2026 at 2:07 PM Dennis Huo <dh...@apache.org> wrote:
> >
> > > Thanks Dmitri, that does help highlight the the concern. Just to make
> > sure
> > > I understand what you have in mind:
> > >
> > > - Shares API defines only shares and the objects placed in shares, with
> > no
> > > Consumer or Listing
> > > - Controlling who can read shares is only ordinary Polaris authz, e.g.
> > > assignPrincipalRoletoPrincipal or
> > addGrantOnSecurableToPrincipalRole(share,
> > > principalRole)
> > > - The partner organization consuming the share is just a principal,
> which
> > > could either be Polaris-managed or from an external IdP
> > >
> > > Is that correct?
> > >
> > > On Wed, Oct 21, 2026 at 4:34 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Dennis,
> > > >
> > > > Let me try to restate my concern from another angle. Apologies that I
> > was
> > > > not very clear in my previous messages.
> > > >
> > > > Currently the create listing doc says: "Create a listing, which
> grants
> > an
> > > > external consumer access to this share".
> > > >
> > > > So, this binds authorization decisions about a Share (Listing) to
> > > > ExternalConsumer.
> > > >
> > > > In IRC use cases a PolarisPrincipal is established by authenticating
> > the
> > > > client request (either via internal JWT or via external IdP),
> > > Subsequently,
> > > > the PolarisPrincipal is passed to PolarisAuthorizer for making
> > > > authorization decisions. The latter part can be external too (e.g.
> OPA
> > /
> > > > Ranger).
> > > >
> > > > (Side note: the "external" work in ExternalConsumer is actually quite
> > > > confusing with respect to prior usage of that word in external IdP
> use
> > > > cases.)
> > > >
> > > > In my view, the ExternalConsumer and related concepts in current
> Shares
> > > API
> > > > do not align with the external IdP and Authorizer workflows.
> > > >
> > > > IIRC, actual access to shared tables is going to use the IRC API,
> > right?
> > > > That access path in existing Polaris use cases maps well to external
> > IdP
> > > > and external authorizers.
> > > >
> > > > Why does the Shares API need to define a "Consumer" as a separate
> > entity
> > > > and narrow authorization decisions to its relationship with Listings?
> > > >
> > > > I believe authentication and authorization options should remain as
> > wide
> > > as
> > > > possible because these matters are usually addressed very differently
> > in
> > > > each deployment.
> > > >
> > > > To be clear: I do not mind defining Share "Consumers" in the internal
> > > > Polaris RBAC domain. It can fit well there, I'm only suggesting
> keeping
> > > the
> > > > Shares API itself neutral to Auth N/Z concerns.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Wed, Oct 21, 2026 at 7:02 PM Dennis Huo <dh...@apache.org> wrote:
> > > >
> > > > > Ah thanks for clarifying - to be clear, the ConsumerCredential is
> > *not*
> > > > > strictly necessary for the Listing in terms of spec expressibility,
> > > only
> > > > > the ExternalConsumer is.
> > > > >
> > > > > An ExternalConsumer doesn't need to be backed by a
> > ConsumerCredential;
> > > > the
> > > > > ExternalConsumer is an abstract Sharing-specific pointer to
> whatever
> > > does
> > > > > the indirection in the sharing implementation.
> > > > >
> > > > > Responded to your github comment - I'll split anything that touches
> > > > > "ConsumerCredential" into a separate yaml file so it's easier to
> see
> > > the
> > > > > core auth-agnostic sharing spec in one file, with the more
> > > > > implementation-specific decorations around ConsumerCredential in an
> > > > > auxiliary file.
> > > > >
> > > > > I suppose another option is to even split it into separate PRs (in
> > > > addition
> > > > > to separate yaml files?). Then we can easily see one commit that's
> > just
> > > > the
> > > > > pure abstract auth-agnostic API, that in theory private Polaris
> > > > > implementations could already take and implement on external-IdP
> > > > > architectures to support the same agnostic spec.
> > > > >
> > > > > On Wed, Oct 21, 2026 at 1:54 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Dennis,
> > > > > >
> > > > > > While I still support the idea of splitting OpenAPI yaml files
> and
> > > URI
> > > > > > bases between the Management and Sharing APIs, I do not think it
> is
> > > > > tightly
> > > > > > coupled with my proposal for the authentication-neural Sharing
> API
> > > > spec.
> > > > > >
> > > > > > To give an example, currently [5446] CreateListingRequest
> includes
> > a
> > > > > > "consumerName". This immediately binds the Listing definition to
> > > > > > an ExternalConsumer and by extension to ConsumerCredential.
> > > > > >
> > > > > > ConsumerCredential is naturally part of the authentication
> process
> > > > during
> > > > > > Share access. I believe ConsumerCredential (and related entities)
> > is
> > > > > worth
> > > > > > isolating from the Shares API to increase interoperability.
> > > > > >
> > > > > > Implementations that choose to rely on ConsumerCredential for
> Share
> > > > > access
> > > > > > can still do so. However, ConsumerCredential will not be a
> > > requirement
> > > > > for
> > > > > > all possible Share API implementations.
> > > > > >
> > > > > > This can work with any OpenAPI yaml and URI paths layout.
> > > > > >
> > > > > > I'd like to resolve this aspect before we go back to URI paths
> and
> > > YAML
> > > > > > file locations. Actually, I do not feel too strongly about path
> > > > > prefixes. I
> > > > > > do believe separate prefixes make implementations easier in many
> > > > aspects
> > > > > > (including network setup), but interleaving those paths with the
> > > > > Management
> > > > > > API is not a blocker, from my POV, only an inconvenience.
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > [5446] https://github.com/apache/polaris/pull/5446
> > > > > >
> > > > > > On Wed, Oct 21, 2026 at 3:19 PM Dennis Huo <dh...@apache.org>
> > wrote:
> > > > > >
> > > > > > > Yeah, that's a good topic to discuss; I guess it touches on our
> > > core
> > > > > > > (unresolved?) conventions on how we organize API url-prefixes
> and
> > > > yaml
> > > > > > > files. Reposting my response to the github comment as well:
> > > > > > >
> > > > > > > Right, I think we could certainly split the API routes and
> > > > > > yaml-definitions
> > > > > > > even further if we want to ensure a well-defined
> auth-detail-free
> > > > core,
> > > > > > but
> > > > > > > for the standalone-Polaris use case we'd still need to define
> it
> > > > > > > *somewhere*
> > > > > > > under a Sharing-specific surface, right?
> > > > > > >
> > > > > > > @dimas-b <https://github.com/dimas-b> Were you thinking of
> just
> > > > > > splitting
> > > > > > > the yaml file but keeping the share-management root, or adding
> a
> > > > third
> > > > > > > sharing prefix like
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials...
> > > > > > > ?
> > > > > > >
> > > > > > > Or just yaml-file split would be keeping
> > > > > > >
> > /api/shares-management/v1/external-consumers/{name}/credentials...
> > > > but
> > > > > > > putting it in a separate yaml file to make it easy for
> > > > service-runners
> > > > > to
> > > > > > > choose not to support that whole yaml file's concrete APIs.
> > > > > > >
> > > > > > > On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > Re: the shating API PR [5446], I'm repeating my GH comment
> here
> > > for
> > > > > > > > visibility.
> > > > > > > >
> > > > > > > > I'd like the API to be neutral from the authentication
> > > perspective.
> > > > > > > >
> > > > > > > > This is how we handle the IRC spec in Polaris. The API
> declares
> > > > that
> > > > > > > > clients need to authenticate, but specific details about
> > > > principals,
> > > > > > > > clients and secrets are delegated to the server.
> > > > > > > >
> > > > > > > > I think this is a sound approach for the sharing API too.
> > > Specific
> > > > > > > details
> > > > > > > > related to native Polaris principals (ConsumerCredential,
> etc.)
> > > can
> > > > > > > > probably remain in the Management API.
> > > > > > > >
> > > > > > > > WDYT?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > [5446] https://github.com/apache/polaris/pull/5446
> > > > > > > >
> > > > > > > > On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré <
> > > > > jb@nanthrax.net>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi
> > > > > > > > >
> > > > > > > > > As I said during the community meeting, let's focus on the
> > API
> > > PR
> > > > > > > > > first. We will build from there.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > JB
> > > > > > > > >
> > > > > > > > > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <dhuo@apache.org
> >
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > +1, thanks for the discussion everyone!
> > > > > > > > > >
> > > > > > > > > > Yes, I meant to flip it to "ready to review" earlier;
> > flipped
> > > > it
> > > > > > now.
> > > > > > > > I'm
> > > > > > > > > > still working through the existing set of comments and
> may
> > > need
> > > > > to
> > > > > > > come
> > > > > > > > > > back to it in the next few days before getting proper
> > > replies,
> > > > > but
> > > > > > > feel
> > > > > > > > > > free to add additional comments.
> > > > > > > > > >
> > > > > > > > > > Recapping the community sync discussion re: Dmitri's last
> > > > > message,
> > > > > > > I'm
> > > > > > > > > > fully aligned on building the sharing on top of existing
> > > authz
> > > > > > > > > mechanisms,
> > > > > > > > > > instead of being a new mechanism, and indeed external IdP
> > is
> > > > > > intended
> > > > > > > > to
> > > > > > > > > be
> > > > > > > > > > a first-class supported use case.
> > > > > > > > > >
> > > > > > > > > > The latest distinction is that the ExternalConsumer isn't
> > > > > > > fundamentally
> > > > > > > > > > identified by any internal PolarisPrincipal, but rather
> > that
> > > it
> > > > > > > serves
> > > > > > > > > as a
> > > > > > > > > > wrapper that references some identity, whether that
> > identity
> > > is
> > > > > > > defined
> > > > > > > > > in
> > > > > > > > > > an external IdP without any latent in-Polaris
> > representation
> > > or
> > > > > if
> > > > > > > > it's a
> > > > > > > > > > hidden PolarisPrincipal (for the Polaris-standalone RBAC
> > use
> > > > > case).
> > > > > > > > > >
> > > > > > > > > > The sharing endpoint via ExternalConsumer is the only
> layer
> > > of
> > > > > > > > > indirection
> > > > > > > > > > at the API entry-point, but then once the set of
> > > > > > activePrincipalRoles
> > > > > > > > is
> > > > > > > > > > resolved, the underlying authz engine doesn't behave any
> > > > > > differently
> > > > > > > > from
> > > > > > > > > > normal Polaris authz.
> > > > > > > > > >
> > > > > > > > > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <
> > > flyrain000@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks everyone for the discussion!
> > > > > > > > > > >
> > > > > > > > > > > As a next step discussed in today's community sync, we
> > will
> > > > > start
> > > > > > > > > reviewing
> > > > > > > > > > > the spec change.
> > > > > > > > > > >
> > > > > > > > > > > Hi Dennis, is this PR ready for review?
> > > > > > > > > > > https://github.com/apache/polaris/pull/5446
> > > > > > > > > > >
> > > > > > > > > > > Yufei
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <
> > > > > > > > travismbowen@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for this discussion Dennis!
> > > > > > > > > > > >
> > > > > > > > > > > > Wanted to weigh in with my thoughts and questions.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Do you prefer /api/shares-management/v1/
> > > > > > > > > > > > > instead of adding to /api/management/v1?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > While I can see the argument for a share specific
> > > > management
> > > > > > > > endpoint
> > > > > > > > > > > there
> > > > > > > > > > > > doesn't seem to be a clear obvious choice from what I
> > can
> > > > see
> > > > > > - I
> > > > > > > > > lean
> > > > > > > > > > > > towards /api/management as being the central place
> for
> > > all
> > > > > the
> > > > > > > > > > > management -
> > > > > > > > > > > > share or not. As long as we're careful not to leak
> > share
> > > > > types
> > > > > > or
> > > > > > > > non
> > > > > > > > > > > share
> > > > > > > > > > > > types into the wrong api (which could be helped by
> > having
> > > > > > > separate
> > > > > > > > > yamls)
> > > > > > > > > > > > then it feels reasonable to have all under the
> > management
> > > > > api.
> > > > > > > > > > > >
> > > > > > > > > > > > Also went through the doc and PR and made several
> > > comments
> > > > as
> > > > > > > well.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <
> > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Prithvi mentioned ExternalConsumer and principals.
> > I'd
> > > > like
> > > > > > to
> > > > > > > > > > > > > reinforce the need to further discuss "consumers"
> and
> > > > > > > principals
> > > > > > > > > and
> > > > > > > > > > > how
> > > > > > > > > > > > > they relate to IdP and Polaris Authentication.
> > > > > > > > > > > > >
> > > > > > > > > > > > > From my POV, share access control should not be a
> new
> > > > > > mechanism
> > > > > > > > but
> > > > > > > > > > > > should
> > > > > > > > > > > > > build on top of the existing extensible
> > authentication
> > > /
> > > > > > > > > authorization
> > > > > > > > > > > > > framework. Supporting external IdP for shares would
> > be
> > > a
> > > > > > > valuable
> > > > > > > > > > > feature
> > > > > > > > > > > > > IMHO.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Side note: I'm posting comments on PR 5446 too.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <
> > > > > > > > > prithvisivasankar@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you for reviving this, and for the concrete
> > > spec
> > > > in
> > > > > > PR
> > > > > > > > > 5446..
> > > > > > > > > > > the
> > > > > > > > > > > > > > journeys doc was especially helpful for seeing
> the
> > > > > > end-to-end
> > > > > > > > > flow.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I agree this is worth doing, and that v1 is more
> > > than a
> > > > > > > > tutorial
> > > > > > > > > on
> > > > > > > > > > > > > today's
> > > > > > > > > > > > > > RBAC. A partner principal on /api/catalog is
> > already
> > > > > > > possible;
> > > > > > > > > what
> > > > > > > > > > > > seems
> > > > > > > > > > > > > > valuable is a first-class share (enumerated
> > members,
> > > a
> > > > > > > > restricted
> > > > > > > > > > > > > consumer,
> > > > > > > > > > > > > > a listing as the binding and audit unit) plus a
> > > > > segregated,
> > > > > > > > > read-only
> > > > > > > > > > > > > > Iceberg REST surface with credential vending.
> Stock
> > > > > Iceberg
> > > > > > > > REST
> > > > > > > > > as
> > > > > > > > > > > the
> > > > > > > > > > > > > > consumer protocol, and keeping Flight and
> > > > > > Polaris-to-Polaris
> > > > > > > > > > > federation
> > > > > > > > > > > > > out
> > > > > > > > > > > > > > of v1, all sound right to me.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I don't have a strong view yet on the URL layout
> > > > > > (management
> > > > > > > vs
> > > > > > > > > > > shares
> > > > > > > > > > > > > > prefix, nested vs top-level, prefix keyed by
> share
> > vs
> > > > > > > listing)
> > > > > > > > > and am
> > > > > > > > > > > > > happy
> > > > > > > > > > > > > > to follow whatever the community prefers there.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > A few things I wanted to check on the spec
> itself,
> > > > > several
> > > > > > of
> > > > > > > > > these
> > > > > > > > > > > > echo
> > > > > > > > > > > > > > points already raised on the list:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Membership as source of truth. If a hidden
> > catalog
> > > > > role
> > > > > > is
> > > > > > > > > > > > > independently
> > > > > > > > > > > > > > grantable, a share could be widened without going
> > > > through
> > > > > > the
> > > > > > > > > share
> > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > Would it make sense to say the data plane
> > authorizes
> > > > > > against
> > > > > > > > > > > > membership +
> > > > > > > > > > > > > > listing, and that any backing roles are not
> > > > user-visible
> > > > > /
> > > > > > > not
> > > > > > > > > > > > > > independently grantable?
> > > > > > > > > > > > > > 2. ExternalConsumer and principals. The Aug 7
> recap
> > > > > already
> > > > > > > > > noted we
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > not assume a PrincipalEntity behind every
> > > > > ExternalConsumer.
> > > > > > > The
> > > > > > > > > > > > journeys
> > > > > > > > > > > > > > still describe granting the consumer a share
> > > principal
> > > > > > role,
> > > > > > > I
> > > > > > > > > assume
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > is an implementation sketch for v1
> > client_id/secret,
> > > > not
> > > > > > the
> > > > > > > > > public
> > > > > > > > > > > > > model?
> > > > > > > > > > > > > > 3. Listing expiration. JB's original note was
> > "share
> > > > > these
> > > > > > > > tables
> > > > > > > > > > > with
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > consumer until this date." Credential expiry is a
> > bit
> > > > > > > > different.
> > > > > > > > > > > Would
> > > > > > > > > > > > a
> > > > > > > > > > > > > > listing-level expiration (and maybe
> > ACTIVE/SUSPENDED)
> > > > be
> > > > > in
> > > > > > > > > scope for
> > > > > > > > > > > > v1?
> > > > > > > > > > > > > > 4. Time-travel. As raised in the community sync,
> > > > sharing
> > > > > > via
> > > > > > > > > Iceberg
> > > > > > > > > > > > REST
> > > > > > > > > > > > > > means historical snapshots remain readable.
> Totally
> > > > fine
> > > > > > for
> > > > > > > v1
> > > > > > > > > to
> > > > > > > > > > > > > support
> > > > > > > > > > > > > > only full history, would an explicit field on the
> > > share
> > > > > > help
> > > > > > > > so a
> > > > > > > > > > > later
> > > > > > > > > > > > > > CURRENT_ONLY option is not a breaking surprise?
> > > > > > > > > > > > > > 5. Consumer data plane. The current file is
> > > > control-plane
> > > > > > > only,
> > > > > > > > > which
> > > > > > > > > > > > is
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > good first step. It might help to list the
> Iceberg
> > > > routes
> > > > > > > that
> > > > > > > > > are
> > > > > > > > > > > > > mounted,
> > > > > > > > > > > > > > that writes 404, that non-members 404, and that
> > > /config
> > > > > > does
> > > > > > > > not
> > > > > > > > > > > return
> > > > > > > > > > > > > > provider catalog properties, so implementations
> > don't
> > > > > > > diverge.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Two smaller notes, if useful: in the J1 journey
> the
> > > > > > consumer
> > > > > > > > > still
> > > > > > > > > > > > posts
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > /api/catalog/v1/oauth/tokens, if partners are not
> > > > > supposed
> > > > > > to
> > > > > > > > > touch
> > > > > > > > > > > the
> > > > > > > > > > > > > > internal catalog API, should the default token
> > > endpoint
> > > > > > live
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > > > > shares
> > > > > > > > > > > > > > surface (or an external IdP)? And on Dmitri's
> point
> > > > about
> > > > > > > share
> > > > > > > > > > > > operators
> > > > > > > > > > > > > > vs catalog operators, a SHARE_MANAGE privilege on
> > the
> > > > > bound
> > > > > > > > > catalog
> > > > > > > > > > > > > > (independent of CATALOG_MANAGE_ACCESS) would make
> > > that
> > > > > work
> > > > > > > on
> > > > > > > > > any
> > > > > > > > > > > URL
> > > > > > > > > > > > > > prefix.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Happy to review the next revision, and thanks
> again
> > > to
> > > > > both
> > > > > > > of
> > > > > > > > > you
> > > > > > > > > > > for
> > > > > > > > > > > > > > driving this.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > Prithvi S
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <
> > > > > dhuo@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As discussed live, looks like we both might
> have
> > > some
> > > > > > > drafts
> > > > > > > > > we can
> > > > > > > > > > > > > > compare
> > > > > > > > > > > > > > > and choose/merge from :)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Here's my current draft spec PR; it opts for my
> > > > > preferred
> > > > > > > > > syntax
> > > > > > > > > > > > > choices
> > > > > > > > > > > > > > > for now (/api/management/ for share-management,
> > > > > > > /api/shares/
> > > > > > > > > for
> > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > data-plane):
> > > > > https://github.com/apache/polaris/pull/5446
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Since we were talking about preferring mdfiles
> > for
> > > > > design
> > > > > > > > > iteration
> > > > > > > > > > > > > > here's
> > > > > > > > > > > > > > > a companion design-addendum which just
> enumerates
> > > > what
> > > > > > the
> > > > > > > > > > > end-to-end
> > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > journeys look like syntactically for each
> > > combination
> > > > > of
> > > > > > > > design
> > > > > > > > > > > > > choices:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Please let me know if anyone prefers a Google
> Doc
> > > > > version
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > > > design-addendum for easier commenting, or if
> the
> > > > > > > > > design-addendum
> > > > > > > > > > > > itself
> > > > > > > > > > > > > > > should also be modeled as a PR on some
> > > free-floating
> > > > > > branch
> > > > > > > > > just to
> > > > > > > > > > > > > > capture
> > > > > > > > > > > > > > > comments.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste
> > > Onofré <
> > > > > > > > > > > jb@nanthrax.net
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I am resuming work on this proposal.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Regarding our recent discussion, I will move
> > > > forward
> > > > > > with
> > > > > > > > > > > opening a
> > > > > > > > > > > > > > > > draft PR for the /api/shares/v1 endpoint to
> > help
> > > > > > > illustrate
> > > > > > > > > the
> > > > > > > > > > > > > > > > design.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I will update this thread once the PR is
> ready.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > JB
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri
> > > Bourlatchkov
> > > > <
> > > > > > > > > > > > dimas@apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Dennis,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for the recap!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I was thinking that share management is
> > > probably
> > > > > > > distinct
> > > > > > > > > from
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > management. A user who already has a
> > > > realm+catalog
> > > > > > may
> > > > > > > > > have an
> > > > > > > > > > > > > option
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > manage shares, but not the catalog itself.
> > > Would
> > > > it
> > > > > > > make
> > > > > > > > > sense
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > the share management API from the general
> > > Polaris
> > > > > > > > > management
> > > > > > > > > > > > API? I
> > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > using a different prefix for URIs.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For example: /api/shares/v1/
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Obviously clients should not make any
> > > assumptions
> > > > > > about
> > > > > > > > the
> > > > > > > > > > > base
> > > > > > > > > > > > > path
> > > > > > > > > > > > > > > > > (/api/shares), which could be different in
> > > > > different
> > > > > > > > > > > deployments
> > > > > > > > > > > > > > (e.g.
> > > > > > > > > > > > > > > > > going through a gateway). Clients must be
> > able
> > > to
> > > > > > take
> > > > > > > > the
> > > > > > > > > base
> > > > > > > > > > > > URI
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > config option. Path segments after /v1/
> will
> > be
> > > > > > > specified
> > > > > > > > > by
> > > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > (Open
> > > > > > > > > > > > > > > > > API yaml).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <
> > > > > > > > dhuo@apache.org
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Just to recap some of the discussion from
> > > > today's
> > > > > > > > > community
> > > > > > > > > > > > sync,
> > > > > > > > > > > > > > > > Dmitri,
> > > > > > > > > > > > > > > > > > Robert, and Alex helped raise a few
> > > > > considerations
> > > > > > to
> > > > > > > > > roll
> > > > > > > > > > > into
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > next
> > > > > > > > > > > > > > > > > > design/prototyping phase:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. Given the way sharing is packaged as a
> > > > > separate
> > > > > > > > > thing, it
> > > > > > > > > > > > may
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > obvious to users that historical
> snapshots
> > > are
> > > > > > > equally
> > > > > > > > > > > > available
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > consumers as the latest snapshot (by
> virtue
> > > of
> > > > > > > sharing
> > > > > > > > > via
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > standard IRC
> > > > > > > > > > > > > > > > > > protocol) - we should make sure to
> document
> > > > that
> > > > > > fact
> > > > > > > > > well,
> > > > > > > > > > > > > > including
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > pitfall that deleting rows by updating a
> > > table
> > > > > > > doesn't
> > > > > > > > > > > > > necessarily
> > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > those rows become inaccessible to
> consumers
> > > > > > > > > > > > > > > > > > 2. Make sure not to hard-code assumptions
> > > about
> > > > > the
> > > > > > > > > existence
> > > > > > > > > > > > of
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > internal PrincipalEntity backing an
> > > > > > ExternalConsumer
> > > > > > > -
> > > > > > > > > even
> > > > > > > > > > > if
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > > MVP implementation uses a nested/hidden
> > > > Principal
> > > > > > of
> > > > > > > > some
> > > > > > > > > > > type
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > holds a
> > > > > > > > > > > > > > > > > > Polaris-managed client_id/client_secret,
> > the
> > > > > > > 3rd-party
> > > > > > > > > IdP
> > > > > > > > > > > use
> > > > > > > > > > > > > > cases
> > > > > > > > > > > > > > > > > > require "implicit" Principals resolved
> via
> > > the
> > > > > > oauth
> > > > > > > > > token.
> > > > > > > > > > > In
> > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > cases
> > > > > > > > > > > > > > > > > > the ExternalConsumer contains the
> metadata
> > > > > > > coordinating
> > > > > > > > > the
> > > > > > > > > > > > > > three-way
> > > > > > > > > > > > > > > > > > handshake between the data-provider,
> > > > > data-consumer,
> > > > > > > and
> > > > > > > > > IdP,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > no PrincipalEntity underlying it.
> > > > > > > > > > > > > > > > > > 3. Longer-term, we may want to tackle
> > > > > > "on-behalf-of"
> > > > > > > > > > > semantics
> > > > > > > > > > > > -
> > > > > > > > > > > > > > > > initially,
> > > > > > > > > > > > > > > > > > the way an ExternalConsumer works is that
> > the
> > > > > > > > > finer-grained
> > > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > share a single ExternalConsumer identity,
> > > just
> > > > > like
> > > > > > > > how a
> > > > > > > > > > > > > FEDERATED
> > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > shares a single service principal in its
> > > > > > > > > ConnectionConfig.
> > > > > > > > > > > But
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > cases the consumer may want some
> end-to-end
> > > > > > > attribution
> > > > > > > > > of
> > > > > > > > > > > > > > > fine-grained
> > > > > > > > > > > > > > > > > > users, like "on-behalf-of: bob @
> > > > > > consumercompany.com
> > > > > > > "
> > > > > > > > > when
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > ExternalConsumer implicit principal is
> > > > something
> > > > > > like
> > > > > > > > > > > > > > > > > > "externalconsumer$consumercompany"
> > > > > > > > > > > > > > > > > > 4. We can iterate more rapidly on a draft
> > > spec
> > > > PR
> > > > > > and
> > > > > > > > > also
> > > > > > > > > > > > expand
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > detailed technical design
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis
> Huo
> > <
> > > > > > > > > dhuo@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks JB for reviving this thread, and
> > > > belated
> > > > > > > > thanks
> > > > > > > > > to
> > > > > > > > > > > all
> > > > > > > > > > > > > > > > reviewers
> > > > > > > > > > > > > > > > > > > who have left feedback in the doc or
> this
> > > > > thread
> > > > > > so
> > > > > > > > > far!
> > > > > > > > > > > And
> > > > > > > > > > > > > > > > apologies
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > the silence on this, it's still been on
> > my
> > > > > radar
> > > > > > > but
> > > > > > > > I
> > > > > > > > > got
> > > > > > > > > > > > > > > > distracted by
> > > > > > > > > > > > > > > > > > > some other work for awhile.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Echoing JB's summary, the proposal is
> > > indeed
> > > > > that
> > > > > > > > we're
> > > > > > > > > > > > mostly
> > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > reusing existing Authz/RBAC machinery,
> > and
> > > > the
> > > > > > > > > translation
> > > > > > > > > > > > > layer
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > sharing concepts sits *on top of* the
> > > > existing
> > > > > > > authz
> > > > > > > > > engine
> > > > > > > > > > > > as
> > > > > > > > > > > > > > much
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > So even if we're adding layers of
> > > indirection
> > > > > on
> > > > > > > > > > > > > > > > ExternalConsumer/Listing
> > > > > > > > > > > > > > > > > > > ->
> > > > Principal/PrincipalRole/CatalogRole+grants,
> > > > > > the
> > > > > > > > > parts
> > > > > > > > > > > that
> > > > > > > > > > > > > end
> > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > the underlying authz just reuse
> existing
> > > > > > Authorizer
> > > > > > > > > logic.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Based on some other offline discussion
> > with
> > > > > > > Dmitri, I
> > > > > > > > > > > > realized
> > > > > > > > > > > > > > why
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > discussion around "requirements" may
> have
> > > > been
> > > > > > > > > confusing:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. My attempts to shorten the doc
> > > > inadvertently
> > > > > > cut
> > > > > > > > out
> > > > > > > > > > > some
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > "abstract requirements" formulations
> > > > > > > > > > > > > > > > > > > 2. The project in itself is somewhat
> more
> > > of
> > > > a
> > > > > > > "form
> > > > > > > > > *as*
> > > > > > > > > > > > > > function"
> > > > > > > > > > > > > > > > > > > feature than *strictly* functional in
> > > nature
> > > > --
> > > > > > for
> > > > > > > > > > > example,
> > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > correctly
> > > > > > > > > > > > > > > > > > > pointed out that "sharing" via the
> > existing
> > > > > > > > > RBAC/Principals
> > > > > > > > > > > > > > syntax
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > already *possible* just by letting a
> > > > cross-org
> > > > > > > > > "consumer"
> > > > > > > > > > > be
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > > > > principal and access your own company's
> > > main
> > > > > > > Polaris
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > endpoints.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > For (1) I added this to the
> Requirements
> > > > > section
> > > > > > in
> > > > > > > > > the doc
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > better
> > > > > > > > > > > > > > > > > > > clarify the abstract form of the data
> > model
> > > > > > > element:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > *A. Sharing Container*: Container
> concept
> > > for
> > > > > > > holding
> > > > > > > > > > > things
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > want to
> > > > > > > > > > > > > > > > > > > share - “ShareEntity”
> > > > > > > > > > > > > > > > > > > *B. Consumer Identity*: Special flavor
> of
> > > > > > > > > principal-like
> > > > > > > > > > > > entity
> > > > > > > > > > > > > > > > > > > representing a consumer relationship -
> > > > > > > > > “ExternalConsumer”
> > > > > > > > > > > > > > > > > > > *C. Map Containers to Consumers*: A way
> > to
> > > > > > specify
> > > > > > > > > letting
> > > > > > > > > > > a
> > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > consumer access a given sharing
> > container -
> > > > > > > “Listing”
> > > > > > > > > > > > > > > > > > > *D. Consumer Endpoint*: A segregated
> API
> > > > > > > access-point
> > > > > > > > > for
> > > > > > > > > > > > > > > consumers -
> > > > > > > > > > > > > > > > > > > “EndpointConfig”
> > > > > > > > > > > > > > > > > > > *E. Objects in Containers*: A way to
> > > specify
> > > > > > which
> > > > > > > > > > > in-catalog
> > > > > > > > > > > > > > > objects
> > > > > > > > > > > > > > > > > > > belong in the big sharing container -
> > > > > > > > “ShareMembership”
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This still bakes in a bit of
> > implementation
> > > > > > detail
> > > > > > > > > since it
> > > > > > > > > > > > > begs
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > question why not just let core
> > RBAC/grants
> > > > > handle
> > > > > > > (C)
> > > > > > > > > and
> > > > > > > > > > > > (E).
> > > > > > > > > > > > > To
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > end,
> > > > > > > > > > > > > > > > > > > and addressing problem (2) of where the
> > > "form
> > > > > > *is*
> > > > > > > > > > > > function", I
> > > > > > > > > > > > > > > > added a
> > > > > > > > > > > > > > > > > > > "Concept Mapping" tab:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think the details of more advanced
> > > features
> > > > > can
> > > > > > > > > still be
> > > > > > > > > > > > > hashed
> > > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > > > incrementally without blocking work on
> an
> > > MVP
> > > > > > (and
> > > > > > > > > > > certainly
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > MVP
> > > > > > > > > > > > > > > > > > scope
> > > > > > > > > > > > > > > > > > > could be trimmed so we can
> > > > launch-and-iterate),
> > > > > > so
> > > > > > > as
> > > > > > > > > long
> > > > > > > > > > > as
> > > > > > > > > > > > > > > there's
> > > > > > > > > > > > > > > > > > > community alignment on the general
> > > direction
> > > > I
> > > > > > > > propose
> > > > > > > > > we
> > > > > > > > > > > > could
> > > > > > > > > > > > > > > move
> > > > > > > > > > > > > > > > > > > forward on basic building blocks. Or if
> > > > > building
> > > > > > > > blocks
> > > > > > > > > > > > > > themselves
> > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > controversial we could still proceed
> with
> > > > some
> > > > > > > > > prototyping
> > > > > > > > > > > > > across
> > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > options.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM
> > > > Jean-Baptiste
> > > > > > > > Onofré <
> > > > > > > > > > > > > > > > jb@nanthrax.net>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >> Hi everyone,
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> I would like to resume the discussion
> > > about
> > > > > Data
> > > > > > > > > Sharing
> > > > > > > > > > > in
> > > > > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > > > >> and propose focusing on three major
> > items
> > > to
> > > > > > help
> > > > > > > us
> > > > > > > > > move
> > > > > > > > > > > > > > forward.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> 1. What is data sharing?
> > > > > > > > > > > > > > > > > > >> In a nutshell, it is an open
> > protocol
> > > > > > proposal
> > > > > > > > > > > allowing
> > > > > > > > > > > > > > > > > > >> organizations to share live data
> without
> > > > > copying
> > > > > > > or
> > > > > > > > > > > > > replication.
> > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > >> main goal is zero-copy sharing
> achieved
> > > > > through
> > > > > > > > shared
> > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > >> authorization.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> On Polaris, this would mean:
> > > > > > > > > > > > > > > > > > >> a. The consumer is authenticated as
> > usual.
> > > > > > > > > > > > > > > > > > >> b. The Polaris server provides the
> > > consumer
> > > > > with
> > > > > > > > > direct
> > > > > > > > > > > > access
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > >> data files (e.g., Parquet, as
> specified
> > in
> > > > the
> > > > > > > > > > > > metadata.json).
> > > > > > > > > > > > > > > > > > >> c. The consumer reads the data files
> > > > directly
> > > > > > > using
> > > > > > > > > their
> > > > > > > > > > > > > > > preferred
> > > > > > > > > > > > > > > > > > >> tool (e.g., a query engine).
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> While Polaris already has most of the
> > > > > necessary
> > > > > > > > > underlying
> > > > > > > > > > > > > > > > > > >> capabilities, we need clear
> > > > branding/packaging
> > > > > > > > around
> > > > > > > > > this
> > > > > > > > > > > > use
> > > > > > > > > > > > > > > case.
> > > > > > > > > > > > > > > > > > >> Specifically, this means:
> > > > > > > > > > > > > > > > > > >> a. A share is a dedicated catalog in
> the
> > > > > Polaris
> > > > > > > > > server,
> > > > > > > > > > > or
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > >> role scoped to specific namespaces.
> > > > > > > > > > > > > > > > > > >> b. A recipient is represented by a
> > > principal
> > > > > > and a
> > > > > > > > > > > principal
> > > > > > > > > > > > > > role.
> > > > > > > > > > > > > > > > > > >> c. A consumer profile consists of the
> > > > catalog
> > > > > > URI
> > > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > > > OAuth2
> > > > > > > > > > > > > > > > > > >> client_id/client_secret.
> > > > > > > > > > > > > > > > > > >> d. We use credential vending
> (temporary
> > > STS
> > > > > > > > > credentials)
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > > > >> access to the data files.
> > > > > > > > > > > > > > > > > > >> e. We use the Polaris IRC layer to
> > expose
> > > > the
> > > > > > > > > metadata and
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > > >> locations/access.
> > > > > > > > > > > > > > > > > > >> f. We can also support
> > Polaris-to-Polaris
> > > > > > sharing
> > > > > > > > via
> > > > > > > > > > > > > > Federation.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Concretely:
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> - On the provider side, we use the
> > > > standard
> > > > > > > > Polaris
> > > > > > > > > RBAC
> > > > > > > > > > > > > model
> > > > > > > > > > > > > > > > > > >> (assigning grants to catalog roles,
> and
> > > > > > assigning
> > > > > > > > > those
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > roles
> > > > > > > > > > > > > > > > > > >> to principal roles):
> > > > > > > > > > > > > > > > > > >> - Create a dedicated principal
> > with
> > > > its
> > > > > > own
> > > > > > > > > > > > > > > > > > client_id/client_secret.
> > > > > > > > > > > > > > > > > > >> - Create a catalog role with
> > > > > > > TABLE_READ_DATA,
> > > > > > > > > > > > > TABLE_LIST,
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > >> NAMESPACE_LIST grants on the
> namespaces
> > to
> > > > be
> > > > > > > shared
> > > > > > > > > (no
> > > > > > > > > > > > write
> > > > > > > > > > > > > > > > > > >> grants).
> > > > > > > > > > > > > > > > > > >> - Create a principal role per
> > > > > destination,
> > > > > > > > > attach
> > > > > > > > > > > the
> > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > >> role, and assign the principal.
> > > > > > > > > > > > > > > > > > >> - The destination engine uses
> > > > loadTable
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > > > > > > > >> "X-Iceberg-Access-Delegation:
> > > > > > vended-credentials"
> > > > > > > > > header.
> > > > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > > >> emits temporary, read-only storage
> > > > > credentials,
> > > > > > > > > allowing
> > > > > > > > > > > the
> > > > > > > > > > > > > > > engine
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > >> read the data files directly from the
> > > > owner's
> > > > > > > bucket
> > > > > > > > > > > without
> > > > > > > > > > > > > > > > accessing
> > > > > > > > > > > > > > > > > > >> the rest of the bucket. This achieves
> > > > > zero-copy
> > > > > > > > > sharing.
> > > > > > > > > > > > > > > > > > >> - On the recipient/consumer side, no
> > > > special
> > > > > > > > > > > configuration
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > >> needed; any IRC client can use it.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> 2. What is data sharing not?
> > > > > > > > > > > > > > > > > > >> I previously considered adding an
> > > Arrow
> > > > > > Flight
> > > > > > > > > > > endpoint
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > >> Polaris, but I no longer think it is a
> > > good
> > > > > idea
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > > initial
> > > > > > > > > > > > > > > > > > >> phase. An Arrow Flight endpoint would
> > > serve
> > > > > the
> > > > > > > data
> > > > > > > > > > > > directly,
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > >> would require Polaris to read the data
> > and
> > > > > > expose
> > > > > > > > it.
> > > > > > > > > I am
> > > > > > > > > > > > > > against
> > > > > > > > > > > > > > > > > > >> embedding a query engine in Polaris,
> as
> > > this
> > > > > > tight
> > > > > > > > > > > coupling
> > > > > > > > > > > > > > feels
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > >> an anti-pattern. While we could use a
> > > > > delegation
> > > > > > > > > service
> > > > > > > > > > > to
> > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > >> Flight endpoint, I think it should be
> > > > > considered
> > > > > > > > for a
> > > > > > > > > > > later
> > > > > > > > > > > > > > > phase,
> > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > >> at all, since a Flight endpoint might
> be
> > > > > better
> > > > > > > > > suited as
> > > > > > > > > > > a
> > > > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > > > >> responsibility and could encourage
> data
> > > > > copying.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> The first phase should focus on
> > packaging
> > > > and
> > > > > > > > > branding the
> > > > > > > > > > > > > > > > > > >> capabilities we already have.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> 3. What could the Data Sharing
> > "package"
> > > > look
> > > > > > > like
> > > > > > > > in
> > > > > > > > > > > > > Polaris?
> > > > > > > > > > > > > > > > > > >> To make Data Sharing obvious and
> > easy
> > > to
> > > > > > use,
> > > > > > > we
> > > > > > > > > > > should
> > > > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > > > >> the missing concept of a "share"—a
> named
> > > > > > > definition
> > > > > > > > > that
> > > > > > > > > > > > users
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > >> list, discover, and define. The goal
> is
> > to
> > > > > > > > > materialize the
> > > > > > > > > > > > > > user's
> > > > > > > > > > > > > > > > > > >> intent ("I want to share these tables
> > with
> > > > > this
> > > > > > > > > consumer
> > > > > > > > > > > > until
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > >> date") into standard Polaris entities
> > > > > > (principal,
> > > > > > > > > > > principal
> > > > > > > > > > > > > > role,
> > > > > > > > > > > > > > > > > > >> catalog role, and grants).
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Therefore, I propose adding a /shares
> > > > endpoint
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > > > >> Management API to package and
> > materialize
> > > > > these
> > > > > > > > share
> > > > > > > > > > > > > > definitions
> > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > >> Polaris entities. This would provide a
> > > > usable,
> > > > > > > > > > > first-version
> > > > > > > > > > > > > > > package
> > > > > > > > > > > > > > > > > > >> for Polaris Data Sharing.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> I hope this addresses Dmitri's
> questions
> > > and
> > > > > > > > provides
> > > > > > > > > a
> > > > > > > > > > > more
> > > > > > > > > > > > > > > > concrete
> > > > > > > > > > > > > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Thoughts?
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Regards,
> > > > > > > > > > > > > > > > > > >> JB
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis
> > > Huo <
> > > > > > > > > > > dhuo@apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >> > Hi All,
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >> > One big emerging enterprise use case
> > > > coming
> > > > > up
> > > > > > > as
> > > > > > > > > more
> > > > > > > > > > > > > people
> > > > > > > > > > > > > > > > > > >> consolidate
> > > > > > > > > > > > > > > > > > >> > Data Lakehouses and Catalogs is
> > > something
> > > > > > > commonly
> > > > > > > > > known
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > "Data
> > > > > > > > > > > > > > > > > > >> Sharing",
> > > > > > > > > > > > > > > > > > >> > an more specifically over the course
> > of
> > > > > > adoption
> > > > > > > > of
> > > > > > > > > open
> > > > > > > > > > > > > table
> > > > > > > > > > > > > > > > formats
> > > > > > > > > > > > > > > > > > >> > "Open Data Sharing".
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >> > Examples of existing managed service
> > > > > > providers'
> > > > > > > > Data
> > > > > > > > > > > > Sharing
> > > > > > > > > > > > > > > > features:
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > https://www.databricks.com/product/delta-sharing
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > >
> > > > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >> > The basic idea is that when you
> share
> > > data
> > > > > > > between
> > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > companies,
> > > > > > > > > > > > > > > > > > >> you
> > > > > > > > > > > > > > > > > > >> > need a first-class
> > governance/management
> > > > > layer
> > > > > > > and
> > > > > > > > > extra
> > > > > > > > > > > > > > > > > > >> bells-and-whistles
> > > > > > > > > > > > > > > > > > >> > that are distinct from just the
> basic
> > > > > > > capabilities
> > > > > > > > > of
> > > > > > > > > > > RBAC
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > >> generalized
> > > > > > > > > > > > > > > > > > >> > access-control (i.e. if you're
> sharing
> > > > > across
> > > > > > > > > > > > > > > partially-untrusted
> > > > > > > > > > > > > > > > org
> > > > > > > > > > > > > > > > > > >> > boundaries, you don't just let the
> > > > consumer
> > > > > > > > > organization
> > > > > > > > > > > > log
> > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > > >> > datalake like one of your own
> > > employees).
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >> > JB and I put together this
> high-level
> > > > > proposal
> > > > > > > for
> > > > > > > > > > > > > supporting
> > > > > > > > > > > > > > > Open
> > > > > > > > > > > > > > > > > > >> Sharing
> > > > > > > > > > > > > > > > > > >> > in Polaris:
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >> > Tentatively, it means adding ~5
> > logical
> > > > data
> > > > > > > model
> > > > > > > > > > > > > constructs,
> > > > > > > > > > > > > > > > some of
> > > > > > > > > > > > > > > > > > >> > which may be a first-class
> > PolarisEntity
> > > > > type,
> > > > > > > > > others
> > > > > > > > > > > > > subtypes
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > >> existing
> > > > > > > > > > > > > > > > > > >> > entities, and others just a nested
> > > > > construct:
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >> > - ShareEntity (would behave
> > similarly
> > > > to
> > > > > a
> > > > > > > > > Catalog)
> > > > > > > > > > > > > > > > > > >> > - ExternalConsumer (mostly
> inherits
> > > > from
> > > > > > > > > Principal)
> > > > > > > > > > > > > > > > > > >> > - Listing (Similar to a "role
> > grant"
> > > > but
> > > > > > has
> > > > > > > > > > > different
> > > > > > > > > > > > > > > > metadata)
> > > > > > > > > > > > > > > > > > >> > - EndpointConfig (nested config
> > under
> > > > > > > Listing)
> > > > > > > > > > > > > > > > > > >> > - ShareMembership (Similar to a
> > > > > "securable
> > > > > > > > > grant" but
> > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > >> metadata)
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >> > Feedback/comments welcome! I'll also
> > > bring
> > > > > it
> > > > > > up
> > > > > > > > for
> > > > > > > > > > > live
> > > > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > >> > there's time in the community sync.
> > > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > > >> > Cheers,
> > > > > > > > > > > > > > > > > > >> > Dennis
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dennis Huo <dh...@apache.org>.
Right, sorry, to clarify, my stated bullets in the last email were my
hypothetical portrayal of what your proposed core set of APIs would be, not
what my current proposal says (so that you can confirm or deny whether
those three bullets convey your ideal state).
It sounds like (2) and (3) match your intent, and (1) is exactly where the
current spec PR differs, since it contains consumers and listings.
For the particular use cases that went into the scoping for the Polaris
Shares proposal, the ExternalConsumer is fairly fundamental to extracting
as a first-class object.
But you're right that there could be a trimmed scoping that allows
implementors to fully abstract away the ExternalConsumer/Listing semantics
instead. In the same way, there's even a fully-trimmed formulation where we
don't implement *any* management APIs at all: Where "Sharing" is just data
providers providing dedicated lightweight Polaris servers, or adding a very
simple config-file-based mapping for what data to make accessible to
dedicated "sharing" Principals or PrincipalRoles.
We could reconcile this by also moving the ExternalConsumer/Listings APIs
to the same "optional" yaml file I moved the other
ConsumerCredential-related endpoints into and keeping the "core Shares API"
limited to what you wanted. Broadly, the two pillars of the proposal are:
1. *What* to share
2. *With whom* to share
The core Shares API would only cover the first, in line with your proposal:
CRUD on Shares and the securables contained in them (listShares,
createShare, getShare, updateShare, deleteShare, listShareMembers,
updateShareMembers).
The optional file would cover the second: ExternalConsumers, Listings, the
endpoint configuration, and credentials for the polaris-standalone case.
Polaris service-runners could then decide if they just want sharing
consumers to live in the exact same identity domain as their other native
Polaris readers, or segregate them by enabling the "With whom" half of the
API.
One thing to discuss for the "core-only" case - today the management API
doesn't syntactically support granting a share to a PrincipalRole, so we'd
need to extend grant types or expose some of the internals of a share's
backing catalog role. I guess we can iteratively refine any necessary API
changes as we discover these in the core-only implementation.
Note that the "with whom" half is necessary for the full "segregation"
reqiurements stated in the original proposal doc, such as preventing
assignment to other roles or accessing the "normal" catalog endpoints,
which is why it can't be *fully* subsumed by only the native Polaris
management API.
It also doesn't conflict with external IdPs: Compare to Databricks'
RECIPIENT for example, which can wrap a Databricks identity, a
provider-issued token, or the recipients own IdP through an OIDC federatoin
policy, and shares are granted uniformly to it in all three cases:
https://docs.databricks.com/aws/en/delta-sharing/create-recipient-oidc-fed
Even in the full API model my PR proposes, we could let the Listing be
something the authorizer looks at alongside other authz conext, so the
pluggable authorizers/OPA/Ranger/etc can integrate as the authoritative
layer.
On Thu, Oct 22, 2026 at 2:00 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi Dennis,
>
> > - Shares API defines only shares and the objects placed in shares, with
> no
> > Consumer or Listing
>
> POST /shares/{shareName}/listings taking a CreateListingRequest with a
> `consumerName` is still part of `spec/polaris-shares-api.yaml`.
>
> I'm kind of confused about the scope of "shares" API vs. other APIs :)
> Where are the boundaries?
>
> Would could be the minimal isoalated API sufficient for managing Shares?
>
> > - Controlling who can read shares is only ordinary Polaris authz, e.g.
> > assignPrincipalRoletoPrincipal or
> addGrantOnSecurableToPrincipalRole(share,
> > principalRole)
>
> Sounds good.
>
> > - The partner organization consuming the share is just a principal, which
> > could either be Polaris-managed or from an external IdP
>
> This also sounds good to me.
>
> I guess the only concern / confusion area is around Listing / Consumer
> (first point above).
>
> Cheers,
> Dmitri.
>
>
> On Thu, Oct 22, 2026 at 2:07 PM Dennis Huo <dh...@apache.org> wrote:
>
> > Thanks Dmitri, that does help highlight the the concern. Just to make
> sure
> > I understand what you have in mind:
> >
> > - Shares API defines only shares and the objects placed in shares, with
> no
> > Consumer or Listing
> > - Controlling who can read shares is only ordinary Polaris authz, e.g.
> > assignPrincipalRoletoPrincipal or
> addGrantOnSecurableToPrincipalRole(share,
> > principalRole)
> > - The partner organization consuming the share is just a principal, which
> > could either be Polaris-managed or from an external IdP
> >
> > Is that correct?
> >
> > On Wed, Oct 21, 2026 at 4:34 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Dennis,
> > >
> > > Let me try to restate my concern from another angle. Apologies that I
> was
> > > not very clear in my previous messages.
> > >
> > > Currently the create listing doc says: "Create a listing, which grants
> an
> > > external consumer access to this share".
> > >
> > > So, this binds authorization decisions about a Share (Listing) to
> > > ExternalConsumer.
> > >
> > > In IRC use cases a PolarisPrincipal is established by authenticating
> the
> > > client request (either via internal JWT or via external IdP),
> > Subsequently,
> > > the PolarisPrincipal is passed to PolarisAuthorizer for making
> > > authorization decisions. The latter part can be external too (e.g. OPA
> /
> > > Ranger).
> > >
> > > (Side note: the "external" work in ExternalConsumer is actually quite
> > > confusing with respect to prior usage of that word in external IdP use
> > > cases.)
> > >
> > > In my view, the ExternalConsumer and related concepts in current Shares
> > API
> > > do not align with the external IdP and Authorizer workflows.
> > >
> > > IIRC, actual access to shared tables is going to use the IRC API,
> right?
> > > That access path in existing Polaris use cases maps well to external
> IdP
> > > and external authorizers.
> > >
> > > Why does the Shares API need to define a "Consumer" as a separate
> entity
> > > and narrow authorization decisions to its relationship with Listings?
> > >
> > > I believe authentication and authorization options should remain as
> wide
> > as
> > > possible because these matters are usually addressed very differently
> in
> > > each deployment.
> > >
> > > To be clear: I do not mind defining Share "Consumers" in the internal
> > > Polaris RBAC domain. It can fit well there, I'm only suggesting keeping
> > the
> > > Shares API itself neutral to Auth N/Z concerns.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Wed, Oct 21, 2026 at 7:02 PM Dennis Huo <dh...@apache.org> wrote:
> > >
> > > > Ah thanks for clarifying - to be clear, the ConsumerCredential is
> *not*
> > > > strictly necessary for the Listing in terms of spec expressibility,
> > only
> > > > the ExternalConsumer is.
> > > >
> > > > An ExternalConsumer doesn't need to be backed by a
> ConsumerCredential;
> > > the
> > > > ExternalConsumer is an abstract Sharing-specific pointer to whatever
> > does
> > > > the indirection in the sharing implementation.
> > > >
> > > > Responded to your github comment - I'll split anything that touches
> > > > "ConsumerCredential" into a separate yaml file so it's easier to see
> > the
> > > > core auth-agnostic sharing spec in one file, with the more
> > > > implementation-specific decorations around ConsumerCredential in an
> > > > auxiliary file.
> > > >
> > > > I suppose another option is to even split it into separate PRs (in
> > > addition
> > > > to separate yaml files?). Then we can easily see one commit that's
> just
> > > the
> > > > pure abstract auth-agnostic API, that in theory private Polaris
> > > > implementations could already take and implement on external-IdP
> > > > architectures to support the same agnostic spec.
> > > >
> > > > On Wed, Oct 21, 2026 at 1:54 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Dennis,
> > > > >
> > > > > While I still support the idea of splitting OpenAPI yaml files and
> > URI
> > > > > bases between the Management and Sharing APIs, I do not think it is
> > > > tightly
> > > > > coupled with my proposal for the authentication-neural Sharing API
> > > spec.
> > > > >
> > > > > To give an example, currently [5446] CreateListingRequest includes
> a
> > > > > "consumerName". This immediately binds the Listing definition to
> > > > > an ExternalConsumer and by extension to ConsumerCredential.
> > > > >
> > > > > ConsumerCredential is naturally part of the authentication process
> > > during
> > > > > Share access. I believe ConsumerCredential (and related entities)
> is
> > > > worth
> > > > > isolating from the Shares API to increase interoperability.
> > > > >
> > > > > Implementations that choose to rely on ConsumerCredential for Share
> > > > access
> > > > > can still do so. However, ConsumerCredential will not be a
> > requirement
> > > > for
> > > > > all possible Share API implementations.
> > > > >
> > > > > This can work with any OpenAPI yaml and URI paths layout.
> > > > >
> > > > > I'd like to resolve this aspect before we go back to URI paths and
> > YAML
> > > > > file locations. Actually, I do not feel too strongly about path
> > > > prefixes. I
> > > > > do believe separate prefixes make implementations easier in many
> > > aspects
> > > > > (including network setup), but interleaving those paths with the
> > > > Management
> > > > > API is not a blocker, from my POV, only an inconvenience.
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > [5446] https://github.com/apache/polaris/pull/5446
> > > > >
> > > > > On Wed, Oct 21, 2026 at 3:19 PM Dennis Huo <dh...@apache.org>
> wrote:
> > > > >
> > > > > > Yeah, that's a good topic to discuss; I guess it touches on our
> > core
> > > > > > (unresolved?) conventions on how we organize API url-prefixes and
> > > yaml
> > > > > > files. Reposting my response to the github comment as well:
> > > > > >
> > > > > > Right, I think we could certainly split the API routes and
> > > > > yaml-definitions
> > > > > > even further if we want to ensure a well-defined auth-detail-free
> > > core,
> > > > > but
> > > > > > for the standalone-Polaris use case we'd still need to define it
> > > > > > *somewhere*
> > > > > > under a Sharing-specific surface, right?
> > > > > >
> > > > > > @dimas-b <https://github.com/dimas-b> Were you thinking of just
> > > > > splitting
> > > > > > the yaml file but keeping the share-management root, or adding a
> > > third
> > > > > > sharing prefix like
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials...
> > > > > > ?
> > > > > >
> > > > > > Or just yaml-file split would be keeping
> > > > > >
> /api/shares-management/v1/external-consumers/{name}/credentials...
> > > but
> > > > > > putting it in a separate yaml file to make it easy for
> > > service-runners
> > > > to
> > > > > > choose not to support that whole yaml file's concrete APIs.
> > > > > >
> > > > > > On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Re: the shating API PR [5446], I'm repeating my GH comment here
> > for
> > > > > > > visibility.
> > > > > > >
> > > > > > > I'd like the API to be neutral from the authentication
> > perspective.
> > > > > > >
> > > > > > > This is how we handle the IRC spec in Polaris. The API declares
> > > that
> > > > > > > clients need to authenticate, but specific details about
> > > principals,
> > > > > > > clients and secrets are delegated to the server.
> > > > > > >
> > > > > > > I think this is a sound approach for the sharing API too.
> > Specific
> > > > > > details
> > > > > > > related to native Polaris principals (ConsumerCredential, etc.)
> > can
> > > > > > > probably remain in the Management API.
> > > > > > >
> > > > > > > WDYT?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > [5446] https://github.com/apache/polaris/pull/5446
> > > > > > >
> > > > > > > On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré <
> > > > jb@nanthrax.net>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi
> > > > > > > >
> > > > > > > > As I said during the community meeting, let's focus on the
> API
> > PR
> > > > > > > > first. We will build from there.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > JB
> > > > > > > >
> > > > > > > > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <dh...@apache.org>
> > > wrote:
> > > > > > > > >
> > > > > > > > > +1, thanks for the discussion everyone!
> > > > > > > > >
> > > > > > > > > Yes, I meant to flip it to "ready to review" earlier;
> flipped
> > > it
> > > > > now.
> > > > > > > I'm
> > > > > > > > > still working through the existing set of comments and may
> > need
> > > > to
> > > > > > come
> > > > > > > > > back to it in the next few days before getting proper
> > replies,
> > > > but
> > > > > > feel
> > > > > > > > > free to add additional comments.
> > > > > > > > >
> > > > > > > > > Recapping the community sync discussion re: Dmitri's last
> > > > message,
> > > > > > I'm
> > > > > > > > > fully aligned on building the sharing on top of existing
> > authz
> > > > > > > > mechanisms,
> > > > > > > > > instead of being a new mechanism, and indeed external IdP
> is
> > > > > intended
> > > > > > > to
> > > > > > > > be
> > > > > > > > > a first-class supported use case.
> > > > > > > > >
> > > > > > > > > The latest distinction is that the ExternalConsumer isn't
> > > > > > fundamentally
> > > > > > > > > identified by any internal PolarisPrincipal, but rather
> that
> > it
> > > > > > serves
> > > > > > > > as a
> > > > > > > > > wrapper that references some identity, whether that
> identity
> > is
> > > > > > defined
> > > > > > > > in
> > > > > > > > > an external IdP without any latent in-Polaris
> representation
> > or
> > > > if
> > > > > > > it's a
> > > > > > > > > hidden PolarisPrincipal (for the Polaris-standalone RBAC
> use
> > > > case).
> > > > > > > > >
> > > > > > > > > The sharing endpoint via ExternalConsumer is the only layer
> > of
> > > > > > > > indirection
> > > > > > > > > at the API entry-point, but then once the set of
> > > > > activePrincipalRoles
> > > > > > > is
> > > > > > > > > resolved, the underlying authz engine doesn't behave any
> > > > > differently
> > > > > > > from
> > > > > > > > > normal Polaris authz.
> > > > > > > > >
> > > > > > > > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <
> > flyrain000@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks everyone for the discussion!
> > > > > > > > > >
> > > > > > > > > > As a next step discussed in today's community sync, we
> will
> > > > start
> > > > > > > > reviewing
> > > > > > > > > > the spec change.
> > > > > > > > > >
> > > > > > > > > > Hi Dennis, is this PR ready for review?
> > > > > > > > > > https://github.com/apache/polaris/pull/5446
> > > > > > > > > >
> > > > > > > > > > Yufei
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <
> > > > > > > travismbowen@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for this discussion Dennis!
> > > > > > > > > > >
> > > > > > > > > > > Wanted to weigh in with my thoughts and questions.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Do you prefer /api/shares-management/v1/
> > > > > > > > > > > > instead of adding to /api/management/v1?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > While I can see the argument for a share specific
> > > management
> > > > > > > endpoint
> > > > > > > > > > there
> > > > > > > > > > > doesn't seem to be a clear obvious choice from what I
> can
> > > see
> > > > > - I
> > > > > > > > lean
> > > > > > > > > > > towards /api/management as being the central place for
> > all
> > > > the
> > > > > > > > > > management -
> > > > > > > > > > > share or not. As long as we're careful not to leak
> share
> > > > types
> > > > > or
> > > > > > > non
> > > > > > > > > > share
> > > > > > > > > > > types into the wrong api (which could be helped by
> having
> > > > > > separate
> > > > > > > > yamls)
> > > > > > > > > > > then it feels reasonable to have all under the
> management
> > > > api.
> > > > > > > > > > >
> > > > > > > > > > > Also went through the doc and PR and made several
> > comments
> > > as
> > > > > > well.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <
> > > > > > > > dimas@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi All,
> > > > > > > > > > > >
> > > > > > > > > > > > Prithvi mentioned ExternalConsumer and principals.
> I'd
> > > like
> > > > > to
> > > > > > > > > > > > reinforce the need to further discuss "consumers" and
> > > > > > principals
> > > > > > > > and
> > > > > > > > > > how
> > > > > > > > > > > > they relate to IdP and Polaris Authentication.
> > > > > > > > > > > >
> > > > > > > > > > > > From my POV, share access control should not be a new
> > > > > mechanism
> > > > > > > but
> > > > > > > > > > > should
> > > > > > > > > > > > build on top of the existing extensible
> authentication
> > /
> > > > > > > > authorization
> > > > > > > > > > > > framework. Supporting external IdP for shares would
> be
> > a
> > > > > > valuable
> > > > > > > > > > feature
> > > > > > > > > > > > IMHO.
> > > > > > > > > > > >
> > > > > > > > > > > > Side note: I'm posting comments on PR 5446 too.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Dmitri.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <
> > > > > > > > prithvisivasankar@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you for reviving this, and for the concrete
> > spec
> > > in
> > > > > PR
> > > > > > > > 5446..
> > > > > > > > > > the
> > > > > > > > > > > > > journeys doc was especially helpful for seeing the
> > > > > end-to-end
> > > > > > > > flow.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I agree this is worth doing, and that v1 is more
> > than a
> > > > > > > tutorial
> > > > > > > > on
> > > > > > > > > > > > today's
> > > > > > > > > > > > > RBAC. A partner principal on /api/catalog is
> already
> > > > > > possible;
> > > > > > > > what
> > > > > > > > > > > seems
> > > > > > > > > > > > > valuable is a first-class share (enumerated
> members,
> > a
> > > > > > > restricted
> > > > > > > > > > > > consumer,
> > > > > > > > > > > > > a listing as the binding and audit unit) plus a
> > > > segregated,
> > > > > > > > read-only
> > > > > > > > > > > > > Iceberg REST surface with credential vending. Stock
> > > > Iceberg
> > > > > > > REST
> > > > > > > > as
> > > > > > > > > > the
> > > > > > > > > > > > > consumer protocol, and keeping Flight and
> > > > > Polaris-to-Polaris
> > > > > > > > > > federation
> > > > > > > > > > > > out
> > > > > > > > > > > > > of v1, all sound right to me.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't have a strong view yet on the URL layout
> > > > > (management
> > > > > > vs
> > > > > > > > > > shares
> > > > > > > > > > > > > prefix, nested vs top-level, prefix keyed by share
> vs
> > > > > > listing)
> > > > > > > > and am
> > > > > > > > > > > > happy
> > > > > > > > > > > > > to follow whatever the community prefers there.
> > > > > > > > > > > > >
> > > > > > > > > > > > > A few things I wanted to check on the spec itself,
> > > > several
> > > > > of
> > > > > > > > these
> > > > > > > > > > > echo
> > > > > > > > > > > > > points already raised on the list:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Membership as source of truth. If a hidden
> catalog
> > > > role
> > > > > is
> > > > > > > > > > > > independently
> > > > > > > > > > > > > grantable, a share could be widened without going
> > > through
> > > > > the
> > > > > > > > share
> > > > > > > > > > > APIs.
> > > > > > > > > > > > > Would it make sense to say the data plane
> authorizes
> > > > > against
> > > > > > > > > > > membership +
> > > > > > > > > > > > > listing, and that any backing roles are not
> > > user-visible
> > > > /
> > > > > > not
> > > > > > > > > > > > > independently grantable?
> > > > > > > > > > > > > 2. ExternalConsumer and principals. The Aug 7 recap
> > > > already
> > > > > > > > noted we
> > > > > > > > > > > > should
> > > > > > > > > > > > > not assume a PrincipalEntity behind every
> > > > ExternalConsumer.
> > > > > > The
> > > > > > > > > > > journeys
> > > > > > > > > > > > > still describe granting the consumer a share
> > principal
> > > > > role,
> > > > > > I
> > > > > > > > assume
> > > > > > > > > > > > that
> > > > > > > > > > > > > is an implementation sketch for v1
> client_id/secret,
> > > not
> > > > > the
> > > > > > > > public
> > > > > > > > > > > > model?
> > > > > > > > > > > > > 3. Listing expiration. JB's original note was
> "share
> > > > these
> > > > > > > tables
> > > > > > > > > > with
> > > > > > > > > > > > this
> > > > > > > > > > > > > consumer until this date." Credential expiry is a
> bit
> > > > > > > different.
> > > > > > > > > > Would
> > > > > > > > > > > a
> > > > > > > > > > > > > listing-level expiration (and maybe
> ACTIVE/SUSPENDED)
> > > be
> > > > in
> > > > > > > > scope for
> > > > > > > > > > > v1?
> > > > > > > > > > > > > 4. Time-travel. As raised in the community sync,
> > > sharing
> > > > > via
> > > > > > > > Iceberg
> > > > > > > > > > > REST
> > > > > > > > > > > > > means historical snapshots remain readable. Totally
> > > fine
> > > > > for
> > > > > > v1
> > > > > > > > to
> > > > > > > > > > > > support
> > > > > > > > > > > > > only full history, would an explicit field on the
> > share
> > > > > help
> > > > > > > so a
> > > > > > > > > > later
> > > > > > > > > > > > > CURRENT_ONLY option is not a breaking surprise?
> > > > > > > > > > > > > 5. Consumer data plane. The current file is
> > > control-plane
> > > > > > only,
> > > > > > > > which
> > > > > > > > > > > is
> > > > > > > > > > > > a
> > > > > > > > > > > > > good first step. It might help to list the Iceberg
> > > routes
> > > > > > that
> > > > > > > > are
> > > > > > > > > > > > mounted,
> > > > > > > > > > > > > that writes 404, that non-members 404, and that
> > /config
> > > > > does
> > > > > > > not
> > > > > > > > > > return
> > > > > > > > > > > > > provider catalog properties, so implementations
> don't
> > > > > > diverge.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Two smaller notes, if useful: in the J1 journey the
> > > > > consumer
> > > > > > > > still
> > > > > > > > > > > posts
> > > > > > > > > > > > to
> > > > > > > > > > > > > /api/catalog/v1/oauth/tokens, if partners are not
> > > > supposed
> > > > > to
> > > > > > > > touch
> > > > > > > > > > the
> > > > > > > > > > > > > internal catalog API, should the default token
> > endpoint
> > > > > live
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > > shares
> > > > > > > > > > > > > surface (or an external IdP)? And on Dmitri's point
> > > about
> > > > > > share
> > > > > > > > > > > operators
> > > > > > > > > > > > > vs catalog operators, a SHARE_MANAGE privilege on
> the
> > > > bound
> > > > > > > > catalog
> > > > > > > > > > > > > (independent of CATALOG_MANAGE_ACCESS) would make
> > that
> > > > work
> > > > > > on
> > > > > > > > any
> > > > > > > > > > URL
> > > > > > > > > > > > > prefix.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Happy to review the next revision, and thanks again
> > to
> > > > both
> > > > > > of
> > > > > > > > you
> > > > > > > > > > for
> > > > > > > > > > > > > driving this.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > Prithvi S
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <
> > > > dhuo@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > As discussed live, looks like we both might have
> > some
> > > > > > drafts
> > > > > > > > we can
> > > > > > > > > > > > > compare
> > > > > > > > > > > > > > and choose/merge from :)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Here's my current draft spec PR; it opts for my
> > > > preferred
> > > > > > > > syntax
> > > > > > > > > > > > choices
> > > > > > > > > > > > > > for now (/api/management/ for share-management,
> > > > > > /api/shares/
> > > > > > > > for
> > > > > > > > > > > > consumer
> > > > > > > > > > > > > > data-plane):
> > > > https://github.com/apache/polaris/pull/5446
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Since we were talking about preferring mdfiles
> for
> > > > design
> > > > > > > > iteration
> > > > > > > > > > > > > here's
> > > > > > > > > > > > > > a companion design-addendum which just enumerates
> > > what
> > > > > the
> > > > > > > > > > end-to-end
> > > > > > > > > > > > > user
> > > > > > > > > > > > > > journeys look like syntactically for each
> > combination
> > > > of
> > > > > > > design
> > > > > > > > > > > > choices:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please let me know if anyone prefers a Google Doc
> > > > version
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > > > design-addendum for easier commenting, or if the
> > > > > > > > design-addendum
> > > > > > > > > > > itself
> > > > > > > > > > > > > > should also be modeled as a PR on some
> > free-floating
> > > > > branch
> > > > > > > > just to
> > > > > > > > > > > > > capture
> > > > > > > > > > > > > > comments.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste
> > Onofré <
> > > > > > > > > > jb@nanthrax.net
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I am resuming work on this proposal.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regarding our recent discussion, I will move
> > > forward
> > > > > with
> > > > > > > > > > opening a
> > > > > > > > > > > > > > > draft PR for the /api/shares/v1 endpoint to
> help
> > > > > > illustrate
> > > > > > > > the
> > > > > > > > > > > > > > > design.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I will update this thread once the PR is ready.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > JB
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri
> > Bourlatchkov
> > > <
> > > > > > > > > > > dimas@apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Dennis,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the recap!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I was thinking that share management is
> > probably
> > > > > > distinct
> > > > > > > > from
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > management. A user who already has a
> > > realm+catalog
> > > > > may
> > > > > > > > have an
> > > > > > > > > > > > option
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > manage shares, but not the catalog itself.
> > Would
> > > it
> > > > > > make
> > > > > > > > sense
> > > > > > > > > > to
> > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > the share management API from the general
> > Polaris
> > > > > > > > management
> > > > > > > > > > > API? I
> > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > using a different prefix for URIs.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For example: /api/shares/v1/
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Obviously clients should not make any
> > assumptions
> > > > > about
> > > > > > > the
> > > > > > > > > > base
> > > > > > > > > > > > path
> > > > > > > > > > > > > > > > (/api/shares), which could be different in
> > > > different
> > > > > > > > > > deployments
> > > > > > > > > > > > > (e.g.
> > > > > > > > > > > > > > > > going through a gateway). Clients must be
> able
> > to
> > > > > take
> > > > > > > the
> > > > > > > > base
> > > > > > > > > > > URI
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > config option. Path segments after /v1/ will
> be
> > > > > > specified
> > > > > > > > by
> > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > (Open
> > > > > > > > > > > > > > > > API yaml).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <
> > > > > > > dhuo@apache.org
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Just to recap some of the discussion from
> > > today's
> > > > > > > > community
> > > > > > > > > > > sync,
> > > > > > > > > > > > > > > Dmitri,
> > > > > > > > > > > > > > > > > Robert, and Alex helped raise a few
> > > > considerations
> > > > > to
> > > > > > > > roll
> > > > > > > > > > into
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > next
> > > > > > > > > > > > > > > > > design/prototyping phase:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. Given the way sharing is packaged as a
> > > > separate
> > > > > > > > thing, it
> > > > > > > > > > > may
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > obvious to users that historical snapshots
> > are
> > > > > > equally
> > > > > > > > > > > available
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > consumers as the latest snapshot (by virtue
> > of
> > > > > > sharing
> > > > > > > > via
> > > > > > > > > > the
> > > > > > > > > > > > > > > standard IRC
> > > > > > > > > > > > > > > > > protocol) - we should make sure to document
> > > that
> > > > > fact
> > > > > > > > well,
> > > > > > > > > > > > > including
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > pitfall that deleting rows by updating a
> > table
> > > > > > doesn't
> > > > > > > > > > > > necessarily
> > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > those rows become inaccessible to consumers
> > > > > > > > > > > > > > > > > 2. Make sure not to hard-code assumptions
> > about
> > > > the
> > > > > > > > existence
> > > > > > > > > > > of
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > internal PrincipalEntity backing an
> > > > > ExternalConsumer
> > > > > > -
> > > > > > > > even
> > > > > > > > > > if
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > MVP implementation uses a nested/hidden
> > > Principal
> > > > > of
> > > > > > > some
> > > > > > > > > > type
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > holds a
> > > > > > > > > > > > > > > > > Polaris-managed client_id/client_secret,
> the
> > > > > > 3rd-party
> > > > > > > > IdP
> > > > > > > > > > use
> > > > > > > > > > > > > cases
> > > > > > > > > > > > > > > > > require "implicit" Principals resolved via
> > the
> > > > > oauth
> > > > > > > > token.
> > > > > > > > > > In
> > > > > > > > > > > > such
> > > > > > > > > > > > > > > cases
> > > > > > > > > > > > > > > > > the ExternalConsumer contains the metadata
> > > > > > coordinating
> > > > > > > > the
> > > > > > > > > > > > > three-way
> > > > > > > > > > > > > > > > > handshake between the data-provider,
> > > > data-consumer,
> > > > > > and
> > > > > > > > IdP,
> > > > > > > > > > > but
> > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > no PrincipalEntity underlying it.
> > > > > > > > > > > > > > > > > 3. Longer-term, we may want to tackle
> > > > > "on-behalf-of"
> > > > > > > > > > semantics
> > > > > > > > > > > -
> > > > > > > > > > > > > > > initially,
> > > > > > > > > > > > > > > > > the way an ExternalConsumer works is that
> the
> > > > > > > > finer-grained
> > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > share a single ExternalConsumer identity,
> > just
> > > > like
> > > > > > > how a
> > > > > > > > > > > > FEDERATED
> > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > shares a single service principal in its
> > > > > > > > ConnectionConfig.
> > > > > > > > > > But
> > > > > > > > > > > in
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > cases the consumer may want some end-to-end
> > > > > > attribution
> > > > > > > > of
> > > > > > > > > > > > > > fine-grained
> > > > > > > > > > > > > > > > > users, like "on-behalf-of: bob @
> > > > > consumercompany.com
> > > > > > "
> > > > > > > > when
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > ExternalConsumer implicit principal is
> > > something
> > > > > like
> > > > > > > > > > > > > > > > > "externalconsumer$consumercompany"
> > > > > > > > > > > > > > > > > 4. We can iterate more rapidly on a draft
> > spec
> > > PR
> > > > > and
> > > > > > > > also
> > > > > > > > > > > expand
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > detailed technical design
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo
> <
> > > > > > > > dhuo@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks JB for reviving this thread, and
> > > belated
> > > > > > > thanks
> > > > > > > > to
> > > > > > > > > > all
> > > > > > > > > > > > > > > reviewers
> > > > > > > > > > > > > > > > > > who have left feedback in the doc or this
> > > > thread
> > > > > so
> > > > > > > > far!
> > > > > > > > > > And
> > > > > > > > > > > > > > > apologies
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > the silence on this, it's still been on
> my
> > > > radar
> > > > > > but
> > > > > > > I
> > > > > > > > got
> > > > > > > > > > > > > > > distracted by
> > > > > > > > > > > > > > > > > > some other work for awhile.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Echoing JB's summary, the proposal is
> > indeed
> > > > that
> > > > > > > we're
> > > > > > > > > > > mostly
> > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > reusing existing Authz/RBAC machinery,
> and
> > > the
> > > > > > > > translation
> > > > > > > > > > > > layer
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > sharing concepts sits *on top of* the
> > > existing
> > > > > > authz
> > > > > > > > engine
> > > > > > > > > > > as
> > > > > > > > > > > > > much
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > So even if we're adding layers of
> > indirection
> > > > on
> > > > > > > > > > > > > > > ExternalConsumer/Listing
> > > > > > > > > > > > > > > > > > ->
> > > Principal/PrincipalRole/CatalogRole+grants,
> > > > > the
> > > > > > > > parts
> > > > > > > > > > that
> > > > > > > > > > > > end
> > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > the underlying authz just reuse existing
> > > > > Authorizer
> > > > > > > > logic.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Based on some other offline discussion
> with
> > > > > > Dmitri, I
> > > > > > > > > > > realized
> > > > > > > > > > > > > why
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > discussion around "requirements" may have
> > > been
> > > > > > > > confusing:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. My attempts to shorten the doc
> > > inadvertently
> > > > > cut
> > > > > > > out
> > > > > > > > > > some
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > "abstract requirements" formulations
> > > > > > > > > > > > > > > > > > 2. The project in itself is somewhat more
> > of
> > > a
> > > > > > "form
> > > > > > > > *as*
> > > > > > > > > > > > > function"
> > > > > > > > > > > > > > > > > > feature than *strictly* functional in
> > nature
> > > --
> > > > > for
> > > > > > > > > > example,
> > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > correctly
> > > > > > > > > > > > > > > > > > pointed out that "sharing" via the
> existing
> > > > > > > > RBAC/Principals
> > > > > > > > > > > > > syntax
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > already *possible* just by letting a
> > > cross-org
> > > > > > > > "consumer"
> > > > > > > > > > be
> > > > > > > > > > > a
> > > > > > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > > > principal and access your own company's
> > main
> > > > > > Polaris
> > > > > > > > > > catalog
> > > > > > > > > > > > > > > endpoints.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For (1) I added this to the Requirements
> > > > section
> > > > > in
> > > > > > > > the doc
> > > > > > > > > > > to
> > > > > > > > > > > > > > better
> > > > > > > > > > > > > > > > > > clarify the abstract form of the data
> model
> > > > > > element:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *A. Sharing Container*: Container concept
> > for
> > > > > > holding
> > > > > > > > > > things
> > > > > > > > > > > > you
> > > > > > > > > > > > > > > want to
> > > > > > > > > > > > > > > > > > share - “ShareEntity”
> > > > > > > > > > > > > > > > > > *B. Consumer Identity*: Special flavor of
> > > > > > > > principal-like
> > > > > > > > > > > entity
> > > > > > > > > > > > > > > > > > representing a consumer relationship -
> > > > > > > > “ExternalConsumer”
> > > > > > > > > > > > > > > > > > *C. Map Containers to Consumers*: A way
> to
> > > > > specify
> > > > > > > > letting
> > > > > > > > > > a
> > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > consumer access a given sharing
> container -
> > > > > > “Listing”
> > > > > > > > > > > > > > > > > > *D. Consumer Endpoint*: A segregated API
> > > > > > access-point
> > > > > > > > for
> > > > > > > > > > > > > > consumers -
> > > > > > > > > > > > > > > > > > “EndpointConfig”
> > > > > > > > > > > > > > > > > > *E. Objects in Containers*: A way to
> > specify
> > > > > which
> > > > > > > > > > in-catalog
> > > > > > > > > > > > > > objects
> > > > > > > > > > > > > > > > > > belong in the big sharing container -
> > > > > > > “ShareMembership”
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This still bakes in a bit of
> implementation
> > > > > detail
> > > > > > > > since it
> > > > > > > > > > > > begs
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > question why not just let core
> RBAC/grants
> > > > handle
> > > > > > (C)
> > > > > > > > and
> > > > > > > > > > > (E).
> > > > > > > > > > > > To
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > end,
> > > > > > > > > > > > > > > > > > and addressing problem (2) of where the
> > "form
> > > > > *is*
> > > > > > > > > > > function", I
> > > > > > > > > > > > > > > added a
> > > > > > > > > > > > > > > > > > "Concept Mapping" tab:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think the details of more advanced
> > features
> > > > can
> > > > > > > > still be
> > > > > > > > > > > > hashed
> > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > > incrementally without blocking work on an
> > MVP
> > > > > (and
> > > > > > > > > > certainly
> > > > > > > > > > > > the
> > > > > > > > > > > > > > MVP
> > > > > > > > > > > > > > > > > scope
> > > > > > > > > > > > > > > > > > could be trimmed so we can
> > > launch-and-iterate),
> > > > > so
> > > > > > as
> > > > > > > > long
> > > > > > > > > > as
> > > > > > > > > > > > > > there's
> > > > > > > > > > > > > > > > > > community alignment on the general
> > direction
> > > I
> > > > > > > propose
> > > > > > > > we
> > > > > > > > > > > could
> > > > > > > > > > > > > > move
> > > > > > > > > > > > > > > > > > forward on basic building blocks. Or if
> > > > building
> > > > > > > blocks
> > > > > > > > > > > > > themselves
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > controversial we could still proceed with
> > > some
> > > > > > > > prototyping
> > > > > > > > > > > > across
> > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > options.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM
> > > Jean-Baptiste
> > > > > > > Onofré <
> > > > > > > > > > > > > > > jb@nanthrax.net>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >> Hi everyone,
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> I would like to resume the discussion
> > about
> > > > Data
> > > > > > > > Sharing
> > > > > > > > > > in
> > > > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > > >> and propose focusing on three major
> items
> > to
> > > > > help
> > > > > > us
> > > > > > > > move
> > > > > > > > > > > > > forward.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> 1. What is data sharing?
> > > > > > > > > > > > > > > > > >> In a nutshell, it is an open
> protocol
> > > > > proposal
> > > > > > > > > > allowing
> > > > > > > > > > > > > > > > > >> organizations to share live data without
> > > > copying
> > > > > > or
> > > > > > > > > > > > replication.
> > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > >> main goal is zero-copy sharing achieved
> > > > through
> > > > > > > shared
> > > > > > > > > > > > metadata
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > >> authorization.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> On Polaris, this would mean:
> > > > > > > > > > > > > > > > > >> a. The consumer is authenticated as
> usual.
> > > > > > > > > > > > > > > > > >> b. The Polaris server provides the
> > consumer
> > > > with
> > > > > > > > direct
> > > > > > > > > > > access
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >> data files (e.g., Parquet, as specified
> in
> > > the
> > > > > > > > > > > metadata.json).
> > > > > > > > > > > > > > > > > >> c. The consumer reads the data files
> > > directly
> > > > > > using
> > > > > > > > their
> > > > > > > > > > > > > > preferred
> > > > > > > > > > > > > > > > > >> tool (e.g., a query engine).
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> While Polaris already has most of the
> > > > necessary
> > > > > > > > underlying
> > > > > > > > > > > > > > > > > >> capabilities, we need clear
> > > branding/packaging
> > > > > > > around
> > > > > > > > this
> > > > > > > > > > > use
> > > > > > > > > > > > > > case.
> > > > > > > > > > > > > > > > > >> Specifically, this means:
> > > > > > > > > > > > > > > > > >> a. A share is a dedicated catalog in the
> > > > Polaris
> > > > > > > > server,
> > > > > > > > > > or
> > > > > > > > > > > a
> > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > >> role scoped to specific namespaces.
> > > > > > > > > > > > > > > > > >> b. A recipient is represented by a
> > principal
> > > > > and a
> > > > > > > > > > principal
> > > > > > > > > > > > > role.
> > > > > > > > > > > > > > > > > >> c. A consumer profile consists of the
> > > catalog
> > > > > URI
> > > > > > > and
> > > > > > > > the
> > > > > > > > > > > > OAuth2
> > > > > > > > > > > > > > > > > >> client_id/client_secret.
> > > > > > > > > > > > > > > > > >> d. We use credential vending (temporary
> > STS
> > > > > > > > credentials)
> > > > > > > > > > for
> > > > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > > >> access to the data files.
> > > > > > > > > > > > > > > > > >> e. We use the Polaris IRC layer to
> expose
> > > the
> > > > > > > > metadata and
> > > > > > > > > > > the
> > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > >> locations/access.
> > > > > > > > > > > > > > > > > >> f. We can also support
> Polaris-to-Polaris
> > > > > sharing
> > > > > > > via
> > > > > > > > > > > > > Federation.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Concretely:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> - On the provider side, we use the
> > > standard
> > > > > > > Polaris
> > > > > > > > RBAC
> > > > > > > > > > > > model
> > > > > > > > > > > > > > > > > >> (assigning grants to catalog roles, and
> > > > > assigning
> > > > > > > > those
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > roles
> > > > > > > > > > > > > > > > > >> to principal roles):
> > > > > > > > > > > > > > > > > >> - Create a dedicated principal
> with
> > > its
> > > > > own
> > > > > > > > > > > > > > > > > client_id/client_secret.
> > > > > > > > > > > > > > > > > >> - Create a catalog role with
> > > > > > TABLE_READ_DATA,
> > > > > > > > > > > > TABLE_LIST,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > >> NAMESPACE_LIST grants on the namespaces
> to
> > > be
> > > > > > shared
> > > > > > > > (no
> > > > > > > > > > > write
> > > > > > > > > > > > > > > > > >> grants).
> > > > > > > > > > > > > > > > > >> - Create a principal role per
> > > > destination,
> > > > > > > > attach
> > > > > > > > > > the
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > >> role, and assign the principal.
> > > > > > > > > > > > > > > > > >> - The destination engine uses
> > > loadTable
> > > > > with
> > > > > > > the
> > > > > > > > > > > > > > > > > >> "X-Iceberg-Access-Delegation:
> > > > > vended-credentials"
> > > > > > > > header.
> > > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > >> emits temporary, read-only storage
> > > > credentials,
> > > > > > > > allowing
> > > > > > > > > > the
> > > > > > > > > > > > > > engine
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > >> read the data files directly from the
> > > owner's
> > > > > > bucket
> > > > > > > > > > without
> > > > > > > > > > > > > > > accessing
> > > > > > > > > > > > > > > > > >> the rest of the bucket. This achieves
> > > > zero-copy
> > > > > > > > sharing.
> > > > > > > > > > > > > > > > > >> - On the recipient/consumer side, no
> > > special
> > > > > > > > > > configuration
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > >> needed; any IRC client can use it.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> 2. What is data sharing not?
> > > > > > > > > > > > > > > > > >> I previously considered adding an
> > Arrow
> > > > > Flight
> > > > > > > > > > endpoint
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > > >> Polaris, but I no longer think it is a
> > good
> > > > idea
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > > initial
> > > > > > > > > > > > > > > > > >> phase. An Arrow Flight endpoint would
> > serve
> > > > the
> > > > > > data
> > > > > > > > > > > directly,
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > >> would require Polaris to read the data
> and
> > > > > expose
> > > > > > > it.
> > > > > > > > I am
> > > > > > > > > > > > > against
> > > > > > > > > > > > > > > > > >> embedding a query engine in Polaris, as
> > this
> > > > > tight
> > > > > > > > > > coupling
> > > > > > > > > > > > > feels
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > >> an anti-pattern. While we could use a
> > > > delegation
> > > > > > > > service
> > > > > > > > > > to
> > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >> Flight endpoint, I think it should be
> > > > considered
> > > > > > > for a
> > > > > > > > > > later
> > > > > > > > > > > > > > phase,
> > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > >> at all, since a Flight endpoint might be
> > > > better
> > > > > > > > suited as
> > > > > > > > > > a
> > > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > > >> responsibility and could encourage data
> > > > copying.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> The first phase should focus on
> packaging
> > > and
> > > > > > > > branding the
> > > > > > > > > > > > > > > > > >> capabilities we already have.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> 3. What could the Data Sharing
> "package"
> > > look
> > > > > > like
> > > > > > > in
> > > > > > > > > > > > Polaris?
> > > > > > > > > > > > > > > > > >> To make Data Sharing obvious and
> easy
> > to
> > > > > use,
> > > > > > we
> > > > > > > > > > should
> > > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > > >> the missing concept of a "share"—a named
> > > > > > definition
> > > > > > > > that
> > > > > > > > > > > users
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > >> list, discover, and define. The goal is
> to
> > > > > > > > materialize the
> > > > > > > > > > > > > user's
> > > > > > > > > > > > > > > > > >> intent ("I want to share these tables
> with
> > > > this
> > > > > > > > consumer
> > > > > > > > > > > until
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > >> date") into standard Polaris entities
> > > > > (principal,
> > > > > > > > > > principal
> > > > > > > > > > > > > role,
> > > > > > > > > > > > > > > > > >> catalog role, and grants).
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Therefore, I propose adding a /shares
> > > endpoint
> > > > > to
> > > > > > > the
> > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > > >> Management API to package and
> materialize
> > > > these
> > > > > > > share
> > > > > > > > > > > > > definitions
> > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > >> Polaris entities. This would provide a
> > > usable,
> > > > > > > > > > first-version
> > > > > > > > > > > > > > package
> > > > > > > > > > > > > > > > > >> for Polaris Data Sharing.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> I hope this addresses Dmitri's questions
> > and
> > > > > > > provides
> > > > > > > > a
> > > > > > > > > > more
> > > > > > > > > > > > > > > concrete
> > > > > > > > > > > > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Thoughts?
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Regards,
> > > > > > > > > > > > > > > > > >> JB
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis
> > Huo <
> > > > > > > > > > dhuo@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> > Hi All,
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> > One big emerging enterprise use case
> > > coming
> > > > up
> > > > > > as
> > > > > > > > more
> > > > > > > > > > > > people
> > > > > > > > > > > > > > > > > >> consolidate
> > > > > > > > > > > > > > > > > >> > Data Lakehouses and Catalogs is
> > something
> > > > > > commonly
> > > > > > > > known
> > > > > > > > > > > as
> > > > > > > > > > > > > > "Data
> > > > > > > > > > > > > > > > > >> Sharing",
> > > > > > > > > > > > > > > > > >> > an more specifically over the course
> of
> > > > > adoption
> > > > > > > of
> > > > > > > > open
> > > > > > > > > > > > table
> > > > > > > > > > > > > > > formats
> > > > > > > > > > > > > > > > > >> > "Open Data Sharing".
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> > Examples of existing managed service
> > > > > providers'
> > > > > > > Data
> > > > > > > > > > > Sharing
> > > > > > > > > > > > > > > features:
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> >
> > > > > > https://www.databricks.com/product/delta-sharing
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > >
> > > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> > The basic idea is that when you share
> > data
> > > > > > between
> > > > > > > > > > > different
> > > > > > > > > > > > > > > > > companies,
> > > > > > > > > > > > > > > > > >> you
> > > > > > > > > > > > > > > > > >> > need a first-class
> governance/management
> > > > layer
> > > > > > and
> > > > > > > > extra
> > > > > > > > > > > > > > > > > >> bells-and-whistles
> > > > > > > > > > > > > > > > > >> > that are distinct from just the basic
> > > > > > capabilities
> > > > > > > > of
> > > > > > > > > > RBAC
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > >> generalized
> > > > > > > > > > > > > > > > > >> > access-control (i.e. if you're sharing
> > > > across
> > > > > > > > > > > > > > partially-untrusted
> > > > > > > > > > > > > > > org
> > > > > > > > > > > > > > > > > >> > boundaries, you don't just let the
> > > consumer
> > > > > > > > organization
> > > > > > > > > > > log
> > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > >> > datalake like one of your own
> > employees).
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> > JB and I put together this high-level
> > > > proposal
> > > > > > for
> > > > > > > > > > > > supporting
> > > > > > > > > > > > > > Open
> > > > > > > > > > > > > > > > > >> Sharing
> > > > > > > > > > > > > > > > > >> > in Polaris:
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> > Tentatively, it means adding ~5
> logical
> > > data
> > > > > > model
> > > > > > > > > > > > constructs,
> > > > > > > > > > > > > > > some of
> > > > > > > > > > > > > > > > > >> > which may be a first-class
> PolarisEntity
> > > > type,
> > > > > > > > others
> > > > > > > > > > > > subtypes
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > >> existing
> > > > > > > > > > > > > > > > > >> > entities, and others just a nested
> > > > construct:
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> > - ShareEntity (would behave
> similarly
> > > to
> > > > a
> > > > > > > > Catalog)
> > > > > > > > > > > > > > > > > >> > - ExternalConsumer (mostly inherits
> > > from
> > > > > > > > Principal)
> > > > > > > > > > > > > > > > > >> > - Listing (Similar to a "role
> grant"
> > > but
> > > > > has
> > > > > > > > > > different
> > > > > > > > > > > > > > > metadata)
> > > > > > > > > > > > > > > > > >> > - EndpointConfig (nested config
> under
> > > > > > Listing)
> > > > > > > > > > > > > > > > > >> > - ShareMembership (Similar to a
> > > > "securable
> > > > > > > > grant" but
> > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > >> metadata)
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> > Feedback/comments welcome! I'll also
> > bring
> > > > it
> > > > > up
> > > > > > > for
> > > > > > > > > > live
> > > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > >> > there's time in the community sync.
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> > Cheers,
> > > > > > > > > > > > > > > > > >> > Dennis
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Dennis,
> - Shares API defines only shares and the objects placed in shares, with no
> Consumer or Listing
POST /shares/{shareName}/listings taking a CreateListingRequest with a
`consumerName` is still part of `spec/polaris-shares-api.yaml`.
I'm kind of confused about the scope of "shares" API vs. other APIs :)
Where are the boundaries?
Would could be the minimal isoalated API sufficient for managing Shares?
> - Controlling who can read shares is only ordinary Polaris authz, e.g.
> assignPrincipalRoletoPrincipal or
addGrantOnSecurableToPrincipalRole(share,
> principalRole)
Sounds good.
> - The partner organization consuming the share is just a principal, which
> could either be Polaris-managed or from an external IdP
This also sounds good to me.
I guess the only concern / confusion area is around Listing / Consumer
(first point above).
Cheers,
Dmitri.
On Thu, Oct 22, 2026 at 2:07 PM Dennis Huo <dh...@apache.org> wrote:
> Thanks Dmitri, that does help highlight the the concern. Just to make sure
> I understand what you have in mind:
>
> - Shares API defines only shares and the objects placed in shares, with no
> Consumer or Listing
> - Controlling who can read shares is only ordinary Polaris authz, e.g.
> assignPrincipalRoletoPrincipal or addGrantOnSecurableToPrincipalRole(share,
> principalRole)
> - The partner organization consuming the share is just a principal, which
> could either be Polaris-managed or from an external IdP
>
> Is that correct?
>
> On Wed, Oct 21, 2026 at 4:34 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Dennis,
> >
> > Let me try to restate my concern from another angle. Apologies that I was
> > not very clear in my previous messages.
> >
> > Currently the create listing doc says: "Create a listing, which grants an
> > external consumer access to this share".
> >
> > So, this binds authorization decisions about a Share (Listing) to
> > ExternalConsumer.
> >
> > In IRC use cases a PolarisPrincipal is established by authenticating the
> > client request (either via internal JWT or via external IdP),
> Subsequently,
> > the PolarisPrincipal is passed to PolarisAuthorizer for making
> > authorization decisions. The latter part can be external too (e.g. OPA /
> > Ranger).
> >
> > (Side note: the "external" work in ExternalConsumer is actually quite
> > confusing with respect to prior usage of that word in external IdP use
> > cases.)
> >
> > In my view, the ExternalConsumer and related concepts in current Shares
> API
> > do not align with the external IdP and Authorizer workflows.
> >
> > IIRC, actual access to shared tables is going to use the IRC API, right?
> > That access path in existing Polaris use cases maps well to external IdP
> > and external authorizers.
> >
> > Why does the Shares API need to define a "Consumer" as a separate entity
> > and narrow authorization decisions to its relationship with Listings?
> >
> > I believe authentication and authorization options should remain as wide
> as
> > possible because these matters are usually addressed very differently in
> > each deployment.
> >
> > To be clear: I do not mind defining Share "Consumers" in the internal
> > Polaris RBAC domain. It can fit well there, I'm only suggesting keeping
> the
> > Shares API itself neutral to Auth N/Z concerns.
> >
> > Cheers,
> > Dmitri.
> >
> > On Wed, Oct 21, 2026 at 7:02 PM Dennis Huo <dh...@apache.org> wrote:
> >
> > > Ah thanks for clarifying - to be clear, the ConsumerCredential is *not*
> > > strictly necessary for the Listing in terms of spec expressibility,
> only
> > > the ExternalConsumer is.
> > >
> > > An ExternalConsumer doesn't need to be backed by a ConsumerCredential;
> > the
> > > ExternalConsumer is an abstract Sharing-specific pointer to whatever
> does
> > > the indirection in the sharing implementation.
> > >
> > > Responded to your github comment - I'll split anything that touches
> > > "ConsumerCredential" into a separate yaml file so it's easier to see
> the
> > > core auth-agnostic sharing spec in one file, with the more
> > > implementation-specific decorations around ConsumerCredential in an
> > > auxiliary file.
> > >
> > > I suppose another option is to even split it into separate PRs (in
> > addition
> > > to separate yaml files?). Then we can easily see one commit that's just
> > the
> > > pure abstract auth-agnostic API, that in theory private Polaris
> > > implementations could already take and implement on external-IdP
> > > architectures to support the same agnostic spec.
> > >
> > > On Wed, Oct 21, 2026 at 1:54 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Dennis,
> > > >
> > > > While I still support the idea of splitting OpenAPI yaml files and
> URI
> > > > bases between the Management and Sharing APIs, I do not think it is
> > > tightly
> > > > coupled with my proposal for the authentication-neural Sharing API
> > spec.
> > > >
> > > > To give an example, currently [5446] CreateListingRequest includes a
> > > > "consumerName". This immediately binds the Listing definition to
> > > > an ExternalConsumer and by extension to ConsumerCredential.
> > > >
> > > > ConsumerCredential is naturally part of the authentication process
> > during
> > > > Share access. I believe ConsumerCredential (and related entities) is
> > > worth
> > > > isolating from the Shares API to increase interoperability.
> > > >
> > > > Implementations that choose to rely on ConsumerCredential for Share
> > > access
> > > > can still do so. However, ConsumerCredential will not be a
> requirement
> > > for
> > > > all possible Share API implementations.
> > > >
> > > > This can work with any OpenAPI yaml and URI paths layout.
> > > >
> > > > I'd like to resolve this aspect before we go back to URI paths and
> YAML
> > > > file locations. Actually, I do not feel too strongly about path
> > > prefixes. I
> > > > do believe separate prefixes make implementations easier in many
> > aspects
> > > > (including network setup), but interleaving those paths with the
> > > Management
> > > > API is not a blocker, from my POV, only an inconvenience.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > [5446] https://github.com/apache/polaris/pull/5446
> > > >
> > > > On Wed, Oct 21, 2026 at 3:19 PM Dennis Huo <dh...@apache.org> wrote:
> > > >
> > > > > Yeah, that's a good topic to discuss; I guess it touches on our
> core
> > > > > (unresolved?) conventions on how we organize API url-prefixes and
> > yaml
> > > > > files. Reposting my response to the github comment as well:
> > > > >
> > > > > Right, I think we could certainly split the API routes and
> > > > yaml-definitions
> > > > > even further if we want to ensure a well-defined auth-detail-free
> > core,
> > > > but
> > > > > for the standalone-Polaris use case we'd still need to define it
> > > > > *somewhere*
> > > > > under a Sharing-specific surface, right?
> > > > >
> > > > > @dimas-b <https://github.com/dimas-b> Were you thinking of just
> > > > splitting
> > > > > the yaml file but keeping the share-management root, or adding a
> > third
> > > > > sharing prefix like
> > > > >
> > > > >
> > > >
> > >
> >
> /api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials...
> > > > > ?
> > > > >
> > > > > Or just yaml-file split would be keeping
> > > > > /api/shares-management/v1/external-consumers/{name}/credentials...
> > but
> > > > > putting it in a separate yaml file to make it easy for
> > service-runners
> > > to
> > > > > choose not to support that whole yaml file's concrete APIs.
> > > > >
> > > > > On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Re: the shating API PR [5446], I'm repeating my GH comment here
> for
> > > > > > visibility.
> > > > > >
> > > > > > I'd like the API to be neutral from the authentication
> perspective.
> > > > > >
> > > > > > This is how we handle the IRC spec in Polaris. The API declares
> > that
> > > > > > clients need to authenticate, but specific details about
> > principals,
> > > > > > clients and secrets are delegated to the server.
> > > > > >
> > > > > > I think this is a sound approach for the sharing API too.
> Specific
> > > > > details
> > > > > > related to native Polaris principals (ConsumerCredential, etc.)
> can
> > > > > > probably remain in the Management API.
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > [5446] https://github.com/apache/polaris/pull/5446
> > > > > >
> > > > > > On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré <
> > > jb@nanthrax.net>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > As I said during the community meeting, let's focus on the API
> PR
> > > > > > > first. We will build from there.
> > > > > > >
> > > > > > > Regards
> > > > > > > JB
> > > > > > >
> > > > > > > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <dh...@apache.org>
> > wrote:
> > > > > > > >
> > > > > > > > +1, thanks for the discussion everyone!
> > > > > > > >
> > > > > > > > Yes, I meant to flip it to "ready to review" earlier; flipped
> > it
> > > > now.
> > > > > > I'm
> > > > > > > > still working through the existing set of comments and may
> need
> > > to
> > > > > come
> > > > > > > > back to it in the next few days before getting proper
> replies,
> > > but
> > > > > feel
> > > > > > > > free to add additional comments.
> > > > > > > >
> > > > > > > > Recapping the community sync discussion re: Dmitri's last
> > > message,
> > > > > I'm
> > > > > > > > fully aligned on building the sharing on top of existing
> authz
> > > > > > > mechanisms,
> > > > > > > > instead of being a new mechanism, and indeed external IdP is
> > > > intended
> > > > > > to
> > > > > > > be
> > > > > > > > a first-class supported use case.
> > > > > > > >
> > > > > > > > The latest distinction is that the ExternalConsumer isn't
> > > > > fundamentally
> > > > > > > > identified by any internal PolarisPrincipal, but rather that
> it
> > > > > serves
> > > > > > > as a
> > > > > > > > wrapper that references some identity, whether that identity
> is
> > > > > defined
> > > > > > > in
> > > > > > > > an external IdP without any latent in-Polaris representation
> or
> > > if
> > > > > > it's a
> > > > > > > > hidden PolarisPrincipal (for the Polaris-standalone RBAC use
> > > case).
> > > > > > > >
> > > > > > > > The sharing endpoint via ExternalConsumer is the only layer
> of
> > > > > > > indirection
> > > > > > > > at the API entry-point, but then once the set of
> > > > activePrincipalRoles
> > > > > > is
> > > > > > > > resolved, the underlying authz engine doesn't behave any
> > > > differently
> > > > > > from
> > > > > > > > normal Polaris authz.
> > > > > > > >
> > > > > > > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <
> flyrain000@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks everyone for the discussion!
> > > > > > > > >
> > > > > > > > > As a next step discussed in today's community sync, we will
> > > start
> > > > > > > reviewing
> > > > > > > > > the spec change.
> > > > > > > > >
> > > > > > > > > Hi Dennis, is this PR ready for review?
> > > > > > > > > https://github.com/apache/polaris/pull/5446
> > > > > > > > >
> > > > > > > > > Yufei
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <
> > > > > > travismbowen@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for this discussion Dennis!
> > > > > > > > > >
> > > > > > > > > > Wanted to weigh in with my thoughts and questions.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Do you prefer /api/shares-management/v1/
> > > > > > > > > > > instead of adding to /api/management/v1?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > While I can see the argument for a share specific
> > management
> > > > > > endpoint
> > > > > > > > > there
> > > > > > > > > > doesn't seem to be a clear obvious choice from what I can
> > see
> > > > - I
> > > > > > > lean
> > > > > > > > > > towards /api/management as being the central place for
> all
> > > the
> > > > > > > > > management -
> > > > > > > > > > share or not. As long as we're careful not to leak share
> > > types
> > > > or
> > > > > > non
> > > > > > > > > share
> > > > > > > > > > types into the wrong api (which could be helped by having
> > > > > separate
> > > > > > > yamls)
> > > > > > > > > > then it feels reasonable to have all under the management
> > > api.
> > > > > > > > > >
> > > > > > > > > > Also went through the doc and PR and made several
> comments
> > as
> > > > > well.
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <
> > > > > > > dimas@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi All,
> > > > > > > > > > >
> > > > > > > > > > > Prithvi mentioned ExternalConsumer and principals. I'd
> > like
> > > > to
> > > > > > > > > > > reinforce the need to further discuss "consumers" and
> > > > > principals
> > > > > > > and
> > > > > > > > > how
> > > > > > > > > > > they relate to IdP and Polaris Authentication.
> > > > > > > > > > >
> > > > > > > > > > > From my POV, share access control should not be a new
> > > > mechanism
> > > > > > but
> > > > > > > > > > should
> > > > > > > > > > > build on top of the existing extensible authentication
> /
> > > > > > > authorization
> > > > > > > > > > > framework. Supporting external IdP for shares would be
> a
> > > > > valuable
> > > > > > > > > feature
> > > > > > > > > > > IMHO.
> > > > > > > > > > >
> > > > > > > > > > > Side note: I'm posting comments on PR 5446 too.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <
> > > > > > > prithvisivasankar@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you for reviving this, and for the concrete
> spec
> > in
> > > > PR
> > > > > > > 5446..
> > > > > > > > > the
> > > > > > > > > > > > journeys doc was especially helpful for seeing the
> > > > end-to-end
> > > > > > > flow.
> > > > > > > > > > > >
> > > > > > > > > > > > I agree this is worth doing, and that v1 is more
> than a
> > > > > > tutorial
> > > > > > > on
> > > > > > > > > > > today's
> > > > > > > > > > > > RBAC. A partner principal on /api/catalog is already
> > > > > possible;
> > > > > > > what
> > > > > > > > > > seems
> > > > > > > > > > > > valuable is a first-class share (enumerated members,
> a
> > > > > > restricted
> > > > > > > > > > > consumer,
> > > > > > > > > > > > a listing as the binding and audit unit) plus a
> > > segregated,
> > > > > > > read-only
> > > > > > > > > > > > Iceberg REST surface with credential vending. Stock
> > > Iceberg
> > > > > > REST
> > > > > > > as
> > > > > > > > > the
> > > > > > > > > > > > consumer protocol, and keeping Flight and
> > > > Polaris-to-Polaris
> > > > > > > > > federation
> > > > > > > > > > > out
> > > > > > > > > > > > of v1, all sound right to me.
> > > > > > > > > > > >
> > > > > > > > > > > > I don't have a strong view yet on the URL layout
> > > > (management
> > > > > vs
> > > > > > > > > shares
> > > > > > > > > > > > prefix, nested vs top-level, prefix keyed by share vs
> > > > > listing)
> > > > > > > and am
> > > > > > > > > > > happy
> > > > > > > > > > > > to follow whatever the community prefers there.
> > > > > > > > > > > >
> > > > > > > > > > > > A few things I wanted to check on the spec itself,
> > > several
> > > > of
> > > > > > > these
> > > > > > > > > > echo
> > > > > > > > > > > > points already raised on the list:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Membership as source of truth. If a hidden catalog
> > > role
> > > > is
> > > > > > > > > > > independently
> > > > > > > > > > > > grantable, a share could be widened without going
> > through
> > > > the
> > > > > > > share
> > > > > > > > > > APIs.
> > > > > > > > > > > > Would it make sense to say the data plane authorizes
> > > > against
> > > > > > > > > > membership +
> > > > > > > > > > > > listing, and that any backing roles are not
> > user-visible
> > > /
> > > > > not
> > > > > > > > > > > > independently grantable?
> > > > > > > > > > > > 2. ExternalConsumer and principals. The Aug 7 recap
> > > already
> > > > > > > noted we
> > > > > > > > > > > should
> > > > > > > > > > > > not assume a PrincipalEntity behind every
> > > ExternalConsumer.
> > > > > The
> > > > > > > > > > journeys
> > > > > > > > > > > > still describe granting the consumer a share
> principal
> > > > role,
> > > > > I
> > > > > > > assume
> > > > > > > > > > > that
> > > > > > > > > > > > is an implementation sketch for v1 client_id/secret,
> > not
> > > > the
> > > > > > > public
> > > > > > > > > > > model?
> > > > > > > > > > > > 3. Listing expiration. JB's original note was "share
> > > these
> > > > > > tables
> > > > > > > > > with
> > > > > > > > > > > this
> > > > > > > > > > > > consumer until this date." Credential expiry is a bit
> > > > > > different.
> > > > > > > > > Would
> > > > > > > > > > a
> > > > > > > > > > > > listing-level expiration (and maybe ACTIVE/SUSPENDED)
> > be
> > > in
> > > > > > > scope for
> > > > > > > > > > v1?
> > > > > > > > > > > > 4. Time-travel. As raised in the community sync,
> > sharing
> > > > via
> > > > > > > Iceberg
> > > > > > > > > > REST
> > > > > > > > > > > > means historical snapshots remain readable. Totally
> > fine
> > > > for
> > > > > v1
> > > > > > > to
> > > > > > > > > > > support
> > > > > > > > > > > > only full history, would an explicit field on the
> share
> > > > help
> > > > > > so a
> > > > > > > > > later
> > > > > > > > > > > > CURRENT_ONLY option is not a breaking surprise?
> > > > > > > > > > > > 5. Consumer data plane. The current file is
> > control-plane
> > > > > only,
> > > > > > > which
> > > > > > > > > > is
> > > > > > > > > > > a
> > > > > > > > > > > > good first step. It might help to list the Iceberg
> > routes
> > > > > that
> > > > > > > are
> > > > > > > > > > > mounted,
> > > > > > > > > > > > that writes 404, that non-members 404, and that
> /config
> > > > does
> > > > > > not
> > > > > > > > > return
> > > > > > > > > > > > provider catalog properties, so implementations don't
> > > > > diverge.
> > > > > > > > > > > >
> > > > > > > > > > > > Two smaller notes, if useful: in the J1 journey the
> > > > consumer
> > > > > > > still
> > > > > > > > > > posts
> > > > > > > > > > > to
> > > > > > > > > > > > /api/catalog/v1/oauth/tokens, if partners are not
> > > supposed
> > > > to
> > > > > > > touch
> > > > > > > > > the
> > > > > > > > > > > > internal catalog API, should the default token
> endpoint
> > > > live
> > > > > on
> > > > > > > the
> > > > > > > > > > > shares
> > > > > > > > > > > > surface (or an external IdP)? And on Dmitri's point
> > about
> > > > > share
> > > > > > > > > > operators
> > > > > > > > > > > > vs catalog operators, a SHARE_MANAGE privilege on the
> > > bound
> > > > > > > catalog
> > > > > > > > > > > > (independent of CATALOG_MANAGE_ACCESS) would make
> that
> > > work
> > > > > on
> > > > > > > any
> > > > > > > > > URL
> > > > > > > > > > > > prefix.
> > > > > > > > > > > >
> > > > > > > > > > > > Happy to review the next revision, and thanks again
> to
> > > both
> > > > > of
> > > > > > > you
> > > > > > > > > for
> > > > > > > > > > > > driving this.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Prithvi S
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <
> > > dhuo@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > As discussed live, looks like we both might have
> some
> > > > > drafts
> > > > > > > we can
> > > > > > > > > > > > compare
> > > > > > > > > > > > > and choose/merge from :)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Here's my current draft spec PR; it opts for my
> > > preferred
> > > > > > > syntax
> > > > > > > > > > > choices
> > > > > > > > > > > > > for now (/api/management/ for share-management,
> > > > > /api/shares/
> > > > > > > for
> > > > > > > > > > > consumer
> > > > > > > > > > > > > data-plane):
> > > https://github.com/apache/polaris/pull/5446
> > > > > > > > > > > > >
> > > > > > > > > > > > > Since we were talking about preferring mdfiles for
> > > design
> > > > > > > iteration
> > > > > > > > > > > > here's
> > > > > > > > > > > > > a companion design-addendum which just enumerates
> > what
> > > > the
> > > > > > > > > end-to-end
> > > > > > > > > > > > user
> > > > > > > > > > > > > journeys look like syntactically for each
> combination
> > > of
> > > > > > design
> > > > > > > > > > > choices:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please let me know if anyone prefers a Google Doc
> > > version
> > > > > of
> > > > > > > the
> > > > > > > > > > > > > design-addendum for easier commenting, or if the
> > > > > > > design-addendum
> > > > > > > > > > itself
> > > > > > > > > > > > > should also be modeled as a PR on some
> free-floating
> > > > branch
> > > > > > > just to
> > > > > > > > > > > > capture
> > > > > > > > > > > > > comments.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste
> Onofré <
> > > > > > > > > jb@nanthrax.net
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I am resuming work on this proposal.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding our recent discussion, I will move
> > forward
> > > > with
> > > > > > > > > opening a
> > > > > > > > > > > > > > draft PR for the /api/shares/v1 endpoint to help
> > > > > illustrate
> > > > > > > the
> > > > > > > > > > > > > > design.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I will update this thread once the PR is ready.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > JB
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri
> Bourlatchkov
> > <
> > > > > > > > > > dimas@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Dennis,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the recap!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I was thinking that share management is
> probably
> > > > > distinct
> > > > > > > from
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > management. A user who already has a
> > realm+catalog
> > > > may
> > > > > > > have an
> > > > > > > > > > > option
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > manage shares, but not the catalog itself.
> Would
> > it
> > > > > make
> > > > > > > sense
> > > > > > > > > to
> > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > the share management API from the general
> Polaris
> > > > > > > management
> > > > > > > > > > API? I
> > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > using a different prefix for URIs.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For example: /api/shares/v1/
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Obviously clients should not make any
> assumptions
> > > > about
> > > > > > the
> > > > > > > > > base
> > > > > > > > > > > path
> > > > > > > > > > > > > > > (/api/shares), which could be different in
> > > different
> > > > > > > > > deployments
> > > > > > > > > > > > (e.g.
> > > > > > > > > > > > > > > going through a gateway). Clients must be able
> to
> > > > take
> > > > > > the
> > > > > > > base
> > > > > > > > > > URI
> > > > > > > > > > > > as
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > config option. Path segments after /v1/ will be
> > > > > specified
> > > > > > > by
> > > > > > > > > > > Polaris
> > > > > > > > > > > > > > (Open
> > > > > > > > > > > > > > > API yaml).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <
> > > > > > dhuo@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Just to recap some of the discussion from
> > today's
> > > > > > > community
> > > > > > > > > > sync,
> > > > > > > > > > > > > > Dmitri,
> > > > > > > > > > > > > > > > Robert, and Alex helped raise a few
> > > considerations
> > > > to
> > > > > > > roll
> > > > > > > > > into
> > > > > > > > > > > the
> > > > > > > > > > > > > > next
> > > > > > > > > > > > > > > > design/prototyping phase:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. Given the way sharing is packaged as a
> > > separate
> > > > > > > thing, it
> > > > > > > > > > may
> > > > > > > > > > > > not
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > obvious to users that historical snapshots
> are
> > > > > equally
> > > > > > > > > > available
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > consumers as the latest snapshot (by virtue
> of
> > > > > sharing
> > > > > > > via
> > > > > > > > > the
> > > > > > > > > > > > > > standard IRC
> > > > > > > > > > > > > > > > protocol) - we should make sure to document
> > that
> > > > fact
> > > > > > > well,
> > > > > > > > > > > > including
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > pitfall that deleting rows by updating a
> table
> > > > > doesn't
> > > > > > > > > > > necessarily
> > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > those rows become inaccessible to consumers
> > > > > > > > > > > > > > > > 2. Make sure not to hard-code assumptions
> about
> > > the
> > > > > > > existence
> > > > > > > > > > of
> > > > > > > > > > > an
> > > > > > > > > > > > > > > > internal PrincipalEntity backing an
> > > > ExternalConsumer
> > > > > -
> > > > > > > even
> > > > > > > > > if
> > > > > > > > > > > the
> > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > MVP implementation uses a nested/hidden
> > Principal
> > > > of
> > > > > > some
> > > > > > > > > type
> > > > > > > > > > > that
> > > > > > > > > > > > > > holds a
> > > > > > > > > > > > > > > > Polaris-managed client_id/client_secret, the
> > > > > 3rd-party
> > > > > > > IdP
> > > > > > > > > use
> > > > > > > > > > > > cases
> > > > > > > > > > > > > > > > require "implicit" Principals resolved via
> the
> > > > oauth
> > > > > > > token.
> > > > > > > > > In
> > > > > > > > > > > such
> > > > > > > > > > > > > > cases
> > > > > > > > > > > > > > > > the ExternalConsumer contains the metadata
> > > > > coordinating
> > > > > > > the
> > > > > > > > > > > > three-way
> > > > > > > > > > > > > > > > handshake between the data-provider,
> > > data-consumer,
> > > > > and
> > > > > > > IdP,
> > > > > > > > > > but
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > no PrincipalEntity underlying it.
> > > > > > > > > > > > > > > > 3. Longer-term, we may want to tackle
> > > > "on-behalf-of"
> > > > > > > > > semantics
> > > > > > > > > > -
> > > > > > > > > > > > > > initially,
> > > > > > > > > > > > > > > > the way an ExternalConsumer works is that the
> > > > > > > finer-grained
> > > > > > > > > > > > consumer
> > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > share a single ExternalConsumer identity,
> just
> > > like
> > > > > > how a
> > > > > > > > > > > FEDERATED
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > shares a single service principal in its
> > > > > > > ConnectionConfig.
> > > > > > > > > But
> > > > > > > > > > in
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > cases the consumer may want some end-to-end
> > > > > attribution
> > > > > > > of
> > > > > > > > > > > > > fine-grained
> > > > > > > > > > > > > > > > users, like "on-behalf-of: bob @
> > > > consumercompany.com
> > > > > "
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > > > > > > > ExternalConsumer implicit principal is
> > something
> > > > like
> > > > > > > > > > > > > > > > "externalconsumer$consumercompany"
> > > > > > > > > > > > > > > > 4. We can iterate more rapidly on a draft
> spec
> > PR
> > > > and
> > > > > > > also
> > > > > > > > > > expand
> > > > > > > > > > > > on
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > detailed technical design
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <
> > > > > > > dhuo@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks JB for reviving this thread, and
> > belated
> > > > > > thanks
> > > > > > > to
> > > > > > > > > all
> > > > > > > > > > > > > > reviewers
> > > > > > > > > > > > > > > > > who have left feedback in the doc or this
> > > thread
> > > > so
> > > > > > > far!
> > > > > > > > > And
> > > > > > > > > > > > > > apologies
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the silence on this, it's still been on my
> > > radar
> > > > > but
> > > > > > I
> > > > > > > got
> > > > > > > > > > > > > > distracted by
> > > > > > > > > > > > > > > > > some other work for awhile.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Echoing JB's summary, the proposal is
> indeed
> > > that
> > > > > > we're
> > > > > > > > > > mostly
> > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > reusing existing Authz/RBAC machinery, and
> > the
> > > > > > > translation
> > > > > > > > > > > layer
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > sharing concepts sits *on top of* the
> > existing
> > > > > authz
> > > > > > > engine
> > > > > > > > > > as
> > > > > > > > > > > > much
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > So even if we're adding layers of
> indirection
> > > on
> > > > > > > > > > > > > > ExternalConsumer/Listing
> > > > > > > > > > > > > > > > > ->
> > Principal/PrincipalRole/CatalogRole+grants,
> > > > the
> > > > > > > parts
> > > > > > > > > that
> > > > > > > > > > > end
> > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > the underlying authz just reuse existing
> > > > Authorizer
> > > > > > > logic.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Based on some other offline discussion with
> > > > > Dmitri, I
> > > > > > > > > > realized
> > > > > > > > > > > > why
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > discussion around "requirements" may have
> > been
> > > > > > > confusing:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. My attempts to shorten the doc
> > inadvertently
> > > > cut
> > > > > > out
> > > > > > > > > some
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > "abstract requirements" formulations
> > > > > > > > > > > > > > > > > 2. The project in itself is somewhat more
> of
> > a
> > > > > "form
> > > > > > > *as*
> > > > > > > > > > > > function"
> > > > > > > > > > > > > > > > > feature than *strictly* functional in
> nature
> > --
> > > > for
> > > > > > > > > example,
> > > > > > > > > > > many
> > > > > > > > > > > > > > > > correctly
> > > > > > > > > > > > > > > > > pointed out that "sharing" via the existing
> > > > > > > RBAC/Principals
> > > > > > > > > > > > syntax
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > already *possible* just by letting a
> > cross-org
> > > > > > > "consumer"
> > > > > > > > > be
> > > > > > > > > > a
> > > > > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > > principal and access your own company's
> main
> > > > > Polaris
> > > > > > > > > catalog
> > > > > > > > > > > > > > endpoints.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For (1) I added this to the Requirements
> > > section
> > > > in
> > > > > > > the doc
> > > > > > > > > > to
> > > > > > > > > > > > > better
> > > > > > > > > > > > > > > > > clarify the abstract form of the data model
> > > > > element:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *A. Sharing Container*: Container concept
> for
> > > > > holding
> > > > > > > > > things
> > > > > > > > > > > you
> > > > > > > > > > > > > > want to
> > > > > > > > > > > > > > > > > share - “ShareEntity”
> > > > > > > > > > > > > > > > > *B. Consumer Identity*: Special flavor of
> > > > > > > principal-like
> > > > > > > > > > entity
> > > > > > > > > > > > > > > > > representing a consumer relationship -
> > > > > > > “ExternalConsumer”
> > > > > > > > > > > > > > > > > *C. Map Containers to Consumers*: A way to
> > > > specify
> > > > > > > letting
> > > > > > > > > a
> > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > consumer access a given sharing container -
> > > > > “Listing”
> > > > > > > > > > > > > > > > > *D. Consumer Endpoint*: A segregated API
> > > > > access-point
> > > > > > > for
> > > > > > > > > > > > > consumers -
> > > > > > > > > > > > > > > > > “EndpointConfig”
> > > > > > > > > > > > > > > > > *E. Objects in Containers*: A way to
> specify
> > > > which
> > > > > > > > > in-catalog
> > > > > > > > > > > > > objects
> > > > > > > > > > > > > > > > > belong in the big sharing container -
> > > > > > “ShareMembership”
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This still bakes in a bit of implementation
> > > > detail
> > > > > > > since it
> > > > > > > > > > > begs
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > question why not just let core RBAC/grants
> > > handle
> > > > > (C)
> > > > > > > and
> > > > > > > > > > (E).
> > > > > > > > > > > To
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > end,
> > > > > > > > > > > > > > > > > and addressing problem (2) of where the
> "form
> > > > *is*
> > > > > > > > > > function", I
> > > > > > > > > > > > > > added a
> > > > > > > > > > > > > > > > > "Concept Mapping" tab:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think the details of more advanced
> features
> > > can
> > > > > > > still be
> > > > > > > > > > > hashed
> > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > incrementally without blocking work on an
> MVP
> > > > (and
> > > > > > > > > certainly
> > > > > > > > > > > the
> > > > > > > > > > > > > MVP
> > > > > > > > > > > > > > > > scope
> > > > > > > > > > > > > > > > > could be trimmed so we can
> > launch-and-iterate),
> > > > so
> > > > > as
> > > > > > > long
> > > > > > > > > as
> > > > > > > > > > > > > there's
> > > > > > > > > > > > > > > > > community alignment on the general
> direction
> > I
> > > > > > propose
> > > > > > > we
> > > > > > > > > > could
> > > > > > > > > > > > > move
> > > > > > > > > > > > > > > > > forward on basic building blocks. Or if
> > > building
> > > > > > blocks
> > > > > > > > > > > > themselves
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > controversial we could still proceed with
> > some
> > > > > > > prototyping
> > > > > > > > > > > across
> > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > options.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM
> > Jean-Baptiste
> > > > > > Onofré <
> > > > > > > > > > > > > > jb@nanthrax.net>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> Hi everyone,
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> I would like to resume the discussion
> about
> > > Data
> > > > > > > Sharing
> > > > > > > > > in
> > > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > >> and propose focusing on three major items
> to
> > > > help
> > > > > us
> > > > > > > move
> > > > > > > > > > > > forward.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> 1. What is data sharing?
> > > > > > > > > > > > > > > > >> In a nutshell, it is an open protocol
> > > > proposal
> > > > > > > > > allowing
> > > > > > > > > > > > > > > > >> organizations to share live data without
> > > copying
> > > > > or
> > > > > > > > > > > replication.
> > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > >> main goal is zero-copy sharing achieved
> > > through
> > > > > > shared
> > > > > > > > > > > metadata
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >> authorization.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> On Polaris, this would mean:
> > > > > > > > > > > > > > > > >> a. The consumer is authenticated as usual.
> > > > > > > > > > > > > > > > >> b. The Polaris server provides the
> consumer
> > > with
> > > > > > > direct
> > > > > > > > > > access
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >> data files (e.g., Parquet, as specified in
> > the
> > > > > > > > > > metadata.json).
> > > > > > > > > > > > > > > > >> c. The consumer reads the data files
> > directly
> > > > > using
> > > > > > > their
> > > > > > > > > > > > > preferred
> > > > > > > > > > > > > > > > >> tool (e.g., a query engine).
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> While Polaris already has most of the
> > > necessary
> > > > > > > underlying
> > > > > > > > > > > > > > > > >> capabilities, we need clear
> > branding/packaging
> > > > > > around
> > > > > > > this
> > > > > > > > > > use
> > > > > > > > > > > > > case.
> > > > > > > > > > > > > > > > >> Specifically, this means:
> > > > > > > > > > > > > > > > >> a. A share is a dedicated catalog in the
> > > Polaris
> > > > > > > server,
> > > > > > > > > or
> > > > > > > > > > a
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > >> role scoped to specific namespaces.
> > > > > > > > > > > > > > > > >> b. A recipient is represented by a
> principal
> > > > and a
> > > > > > > > > principal
> > > > > > > > > > > > role.
> > > > > > > > > > > > > > > > >> c. A consumer profile consists of the
> > catalog
> > > > URI
> > > > > > and
> > > > > > > the
> > > > > > > > > > > OAuth2
> > > > > > > > > > > > > > > > >> client_id/client_secret.
> > > > > > > > > > > > > > > > >> d. We use credential vending (temporary
> STS
> > > > > > > credentials)
> > > > > > > > > for
> > > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > >> access to the data files.
> > > > > > > > > > > > > > > > >> e. We use the Polaris IRC layer to expose
> > the
> > > > > > > metadata and
> > > > > > > > > > the
> > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > >> locations/access.
> > > > > > > > > > > > > > > > >> f. We can also support Polaris-to-Polaris
> > > > sharing
> > > > > > via
> > > > > > > > > > > > Federation.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Concretely:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> - On the provider side, we use the
> > standard
> > > > > > Polaris
> > > > > > > RBAC
> > > > > > > > > > > model
> > > > > > > > > > > > > > > > >> (assigning grants to catalog roles, and
> > > > assigning
> > > > > > > those
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > roles
> > > > > > > > > > > > > > > > >> to principal roles):
> > > > > > > > > > > > > > > > >> - Create a dedicated principal with
> > its
> > > > own
> > > > > > > > > > > > > > > > client_id/client_secret.
> > > > > > > > > > > > > > > > >> - Create a catalog role with
> > > > > TABLE_READ_DATA,
> > > > > > > > > > > TABLE_LIST,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >> NAMESPACE_LIST grants on the namespaces to
> > be
> > > > > shared
> > > > > > > (no
> > > > > > > > > > write
> > > > > > > > > > > > > > > > >> grants).
> > > > > > > > > > > > > > > > >> - Create a principal role per
> > > destination,
> > > > > > > attach
> > > > > > > > > the
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > >> role, and assign the principal.
> > > > > > > > > > > > > > > > >> - The destination engine uses
> > loadTable
> > > > with
> > > > > > the
> > > > > > > > > > > > > > > > >> "X-Iceberg-Access-Delegation:
> > > > vended-credentials"
> > > > > > > header.
> > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > >> emits temporary, read-only storage
> > > credentials,
> > > > > > > allowing
> > > > > > > > > the
> > > > > > > > > > > > > engine
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > >> read the data files directly from the
> > owner's
> > > > > bucket
> > > > > > > > > without
> > > > > > > > > > > > > > accessing
> > > > > > > > > > > > > > > > >> the rest of the bucket. This achieves
> > > zero-copy
> > > > > > > sharing.
> > > > > > > > > > > > > > > > >> - On the recipient/consumer side, no
> > special
> > > > > > > > > configuration
> > > > > > > > > > > is
> > > > > > > > > > > > > > > > >> needed; any IRC client can use it.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> 2. What is data sharing not?
> > > > > > > > > > > > > > > > >> I previously considered adding an
> Arrow
> > > > Flight
> > > > > > > > > endpoint
> > > > > > > > > > to
> > > > > > > > > > > > > > > > >> Polaris, but I no longer think it is a
> good
> > > idea
> > > > > for
> > > > > > > the
> > > > > > > > > > > initial
> > > > > > > > > > > > > > > > >> phase. An Arrow Flight endpoint would
> serve
> > > the
> > > > > data
> > > > > > > > > > directly,
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > >> would require Polaris to read the data and
> > > > expose
> > > > > > it.
> > > > > > > I am
> > > > > > > > > > > > against
> > > > > > > > > > > > > > > > >> embedding a query engine in Polaris, as
> this
> > > > tight
> > > > > > > > > coupling
> > > > > > > > > > > > feels
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > >> an anti-pattern. While we could use a
> > > delegation
> > > > > > > service
> > > > > > > > > to
> > > > > > > > > > > > serve
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >> Flight endpoint, I think it should be
> > > considered
> > > > > > for a
> > > > > > > > > later
> > > > > > > > > > > > > phase,
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > >> at all, since a Flight endpoint might be
> > > better
> > > > > > > suited as
> > > > > > > > > a
> > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > >> responsibility and could encourage data
> > > copying.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> The first phase should focus on packaging
> > and
> > > > > > > branding the
> > > > > > > > > > > > > > > > >> capabilities we already have.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> 3. What could the Data Sharing "package"
> > look
> > > > > like
> > > > > > in
> > > > > > > > > > > Polaris?
> > > > > > > > > > > > > > > > >> To make Data Sharing obvious and easy
> to
> > > > use,
> > > > > we
> > > > > > > > > should
> > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > >> the missing concept of a "share"—a named
> > > > > definition
> > > > > > > that
> > > > > > > > > > users
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > > >> list, discover, and define. The goal is to
> > > > > > > materialize the
> > > > > > > > > > > > user's
> > > > > > > > > > > > > > > > >> intent ("I want to share these tables with
> > > this
> > > > > > > consumer
> > > > > > > > > > until
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > >> date") into standard Polaris entities
> > > > (principal,
> > > > > > > > > principal
> > > > > > > > > > > > role,
> > > > > > > > > > > > > > > > >> catalog role, and grants).
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Therefore, I propose adding a /shares
> > endpoint
> > > > to
> > > > > > the
> > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > >> Management API to package and materialize
> > > these
> > > > > > share
> > > > > > > > > > > > definitions
> > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > >> Polaris entities. This would provide a
> > usable,
> > > > > > > > > first-version
> > > > > > > > > > > > > package
> > > > > > > > > > > > > > > > >> for Polaris Data Sharing.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> I hope this addresses Dmitri's questions
> and
> > > > > > provides
> > > > > > > a
> > > > > > > > > more
> > > > > > > > > > > > > > concrete
> > > > > > > > > > > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Thoughts?
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Regards,
> > > > > > > > > > > > > > > > >> JB
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis
> Huo <
> > > > > > > > > dhuo@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > Hi All,
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > One big emerging enterprise use case
> > coming
> > > up
> > > > > as
> > > > > > > more
> > > > > > > > > > > people
> > > > > > > > > > > > > > > > >> consolidate
> > > > > > > > > > > > > > > > >> > Data Lakehouses and Catalogs is
> something
> > > > > commonly
> > > > > > > known
> > > > > > > > > > as
> > > > > > > > > > > > > "Data
> > > > > > > > > > > > > > > > >> Sharing",
> > > > > > > > > > > > > > > > >> > an more specifically over the course of
> > > > adoption
> > > > > > of
> > > > > > > open
> > > > > > > > > > > table
> > > > > > > > > > > > > > formats
> > > > > > > > > > > > > > > > >> > "Open Data Sharing".
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > Examples of existing managed service
> > > > providers'
> > > > > > Data
> > > > > > > > > > Sharing
> > > > > > > > > > > > > > features:
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> >
> > > > > https://www.databricks.com/product/delta-sharing
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > >
> > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > The basic idea is that when you share
> data
> > > > > between
> > > > > > > > > > different
> > > > > > > > > > > > > > > > companies,
> > > > > > > > > > > > > > > > >> you
> > > > > > > > > > > > > > > > >> > need a first-class governance/management
> > > layer
> > > > > and
> > > > > > > extra
> > > > > > > > > > > > > > > > >> bells-and-whistles
> > > > > > > > > > > > > > > > >> > that are distinct from just the basic
> > > > > capabilities
> > > > > > > of
> > > > > > > > > RBAC
> > > > > > > > > > > or
> > > > > > > > > > > > > > > > >> generalized
> > > > > > > > > > > > > > > > >> > access-control (i.e. if you're sharing
> > > across
> > > > > > > > > > > > > partially-untrusted
> > > > > > > > > > > > > > org
> > > > > > > > > > > > > > > > >> > boundaries, you don't just let the
> > consumer
> > > > > > > organization
> > > > > > > > > > log
> > > > > > > > > > > > > into
> > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > >> > datalake like one of your own
> employees).
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > JB and I put together this high-level
> > > proposal
> > > > > for
> > > > > > > > > > > supporting
> > > > > > > > > > > > > Open
> > > > > > > > > > > > > > > > >> Sharing
> > > > > > > > > > > > > > > > >> > in Polaris:
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > Tentatively, it means adding ~5 logical
> > data
> > > > > model
> > > > > > > > > > > constructs,
> > > > > > > > > > > > > > some of
> > > > > > > > > > > > > > > > >> > which may be a first-class PolarisEntity
> > > type,
> > > > > > > others
> > > > > > > > > > > subtypes
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > >> existing
> > > > > > > > > > > > > > > > >> > entities, and others just a nested
> > > construct:
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > - ShareEntity (would behave similarly
> > to
> > > a
> > > > > > > Catalog)
> > > > > > > > > > > > > > > > >> > - ExternalConsumer (mostly inherits
> > from
> > > > > > > Principal)
> > > > > > > > > > > > > > > > >> > - Listing (Similar to a "role grant"
> > but
> > > > has
> > > > > > > > > different
> > > > > > > > > > > > > > metadata)
> > > > > > > > > > > > > > > > >> > - EndpointConfig (nested config under
> > > > > Listing)
> > > > > > > > > > > > > > > > >> > - ShareMembership (Similar to a
> > > "securable
> > > > > > > grant" but
> > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > >> metadata)
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > Feedback/comments welcome! I'll also
> bring
> > > it
> > > > up
> > > > > > for
> > > > > > > > > live
> > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > >> > there's time in the community sync.
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > Cheers,
> > > > > > > > > > > > > > > > >> > Dennis
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dennis Huo <dh...@apache.org>.
Thanks Dmitri, that does help highlight the the concern. Just to make sure
I understand what you have in mind:
- Shares API defines only shares and the objects placed in shares, with no
Consumer or Listing
- Controlling who can read shares is only ordinary Polaris authz, e.g.
assignPrincipalRoletoPrincipal or addGrantOnSecurableToPrincipalRole(share,
principalRole)
- The partner organization consuming the share is just a principal, which
could either be Polaris-managed or from an external IdP
Is that correct?
On Wed, Oct 21, 2026 at 4:34 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi Dennis,
>
> Let me try to restate my concern from another angle. Apologies that I was
> not very clear in my previous messages.
>
> Currently the create listing doc says: "Create a listing, which grants an
> external consumer access to this share".
>
> So, this binds authorization decisions about a Share (Listing) to
> ExternalConsumer.
>
> In IRC use cases a PolarisPrincipal is established by authenticating the
> client request (either via internal JWT or via external IdP), Subsequently,
> the PolarisPrincipal is passed to PolarisAuthorizer for making
> authorization decisions. The latter part can be external too (e.g. OPA /
> Ranger).
>
> (Side note: the "external" work in ExternalConsumer is actually quite
> confusing with respect to prior usage of that word in external IdP use
> cases.)
>
> In my view, the ExternalConsumer and related concepts in current Shares API
> do not align with the external IdP and Authorizer workflows.
>
> IIRC, actual access to shared tables is going to use the IRC API, right?
> That access path in existing Polaris use cases maps well to external IdP
> and external authorizers.
>
> Why does the Shares API need to define a "Consumer" as a separate entity
> and narrow authorization decisions to its relationship with Listings?
>
> I believe authentication and authorization options should remain as wide as
> possible because these matters are usually addressed very differently in
> each deployment.
>
> To be clear: I do not mind defining Share "Consumers" in the internal
> Polaris RBAC domain. It can fit well there, I'm only suggesting keeping the
> Shares API itself neutral to Auth N/Z concerns.
>
> Cheers,
> Dmitri.
>
> On Wed, Oct 21, 2026 at 7:02 PM Dennis Huo <dh...@apache.org> wrote:
>
> > Ah thanks for clarifying - to be clear, the ConsumerCredential is *not*
> > strictly necessary for the Listing in terms of spec expressibility, only
> > the ExternalConsumer is.
> >
> > An ExternalConsumer doesn't need to be backed by a ConsumerCredential;
> the
> > ExternalConsumer is an abstract Sharing-specific pointer to whatever does
> > the indirection in the sharing implementation.
> >
> > Responded to your github comment - I'll split anything that touches
> > "ConsumerCredential" into a separate yaml file so it's easier to see the
> > core auth-agnostic sharing spec in one file, with the more
> > implementation-specific decorations around ConsumerCredential in an
> > auxiliary file.
> >
> > I suppose another option is to even split it into separate PRs (in
> addition
> > to separate yaml files?). Then we can easily see one commit that's just
> the
> > pure abstract auth-agnostic API, that in theory private Polaris
> > implementations could already take and implement on external-IdP
> > architectures to support the same agnostic spec.
> >
> > On Wed, Oct 21, 2026 at 1:54 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Dennis,
> > >
> > > While I still support the idea of splitting OpenAPI yaml files and URI
> > > bases between the Management and Sharing APIs, I do not think it is
> > tightly
> > > coupled with my proposal for the authentication-neural Sharing API
> spec.
> > >
> > > To give an example, currently [5446] CreateListingRequest includes a
> > > "consumerName". This immediately binds the Listing definition to
> > > an ExternalConsumer and by extension to ConsumerCredential.
> > >
> > > ConsumerCredential is naturally part of the authentication process
> during
> > > Share access. I believe ConsumerCredential (and related entities) is
> > worth
> > > isolating from the Shares API to increase interoperability.
> > >
> > > Implementations that choose to rely on ConsumerCredential for Share
> > access
> > > can still do so. However, ConsumerCredential will not be a requirement
> > for
> > > all possible Share API implementations.
> > >
> > > This can work with any OpenAPI yaml and URI paths layout.
> > >
> > > I'd like to resolve this aspect before we go back to URI paths and YAML
> > > file locations. Actually, I do not feel too strongly about path
> > prefixes. I
> > > do believe separate prefixes make implementations easier in many
> aspects
> > > (including network setup), but interleaving those paths with the
> > Management
> > > API is not a blocker, from my POV, only an inconvenience.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > [5446] https://github.com/apache/polaris/pull/5446
> > >
> > > On Wed, Oct 21, 2026 at 3:19 PM Dennis Huo <dh...@apache.org> wrote:
> > >
> > > > Yeah, that's a good topic to discuss; I guess it touches on our core
> > > > (unresolved?) conventions on how we organize API url-prefixes and
> yaml
> > > > files. Reposting my response to the github comment as well:
> > > >
> > > > Right, I think we could certainly split the API routes and
> > > yaml-definitions
> > > > even further if we want to ensure a well-defined auth-detail-free
> core,
> > > but
> > > > for the standalone-Polaris use case we'd still need to define it
> > > > *somewhere*
> > > > under a Sharing-specific surface, right?
> > > >
> > > > @dimas-b <https://github.com/dimas-b> Were you thinking of just
> > > splitting
> > > > the yaml file but keeping the share-management root, or adding a
> third
> > > > sharing prefix like
> > > >
> > > >
> > >
> >
> /api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials...
> > > > ?
> > > >
> > > > Or just yaml-file split would be keeping
> > > > /api/shares-management/v1/external-consumers/{name}/credentials...
> but
> > > > putting it in a separate yaml file to make it easy for
> service-runners
> > to
> > > > choose not to support that whole yaml file's concrete APIs.
> > > >
> > > > On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Re: the shating API PR [5446], I'm repeating my GH comment here for
> > > > > visibility.
> > > > >
> > > > > I'd like the API to be neutral from the authentication perspective.
> > > > >
> > > > > This is how we handle the IRC spec in Polaris. The API declares
> that
> > > > > clients need to authenticate, but specific details about
> principals,
> > > > > clients and secrets are delegated to the server.
> > > > >
> > > > > I think this is a sound approach for the sharing API too. Specific
> > > > details
> > > > > related to native Polaris principals (ConsumerCredential, etc.) can
> > > > > probably remain in the Management API.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > [5446] https://github.com/apache/polaris/pull/5446
> > > > >
> > > > > On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré <
> > jb@nanthrax.net>
> > > > > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > As I said during the community meeting, let's focus on the API PR
> > > > > > first. We will build from there.
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <dh...@apache.org>
> wrote:
> > > > > > >
> > > > > > > +1, thanks for the discussion everyone!
> > > > > > >
> > > > > > > Yes, I meant to flip it to "ready to review" earlier; flipped
> it
> > > now.
> > > > > I'm
> > > > > > > still working through the existing set of comments and may need
> > to
> > > > come
> > > > > > > back to it in the next few days before getting proper replies,
> > but
> > > > feel
> > > > > > > free to add additional comments.
> > > > > > >
> > > > > > > Recapping the community sync discussion re: Dmitri's last
> > message,
> > > > I'm
> > > > > > > fully aligned on building the sharing on top of existing authz
> > > > > > mechanisms,
> > > > > > > instead of being a new mechanism, and indeed external IdP is
> > > intended
> > > > > to
> > > > > > be
> > > > > > > a first-class supported use case.
> > > > > > >
> > > > > > > The latest distinction is that the ExternalConsumer isn't
> > > > fundamentally
> > > > > > > identified by any internal PolarisPrincipal, but rather that it
> > > > serves
> > > > > > as a
> > > > > > > wrapper that references some identity, whether that identity is
> > > > defined
> > > > > > in
> > > > > > > an external IdP without any latent in-Polaris representation or
> > if
> > > > > it's a
> > > > > > > hidden PolarisPrincipal (for the Polaris-standalone RBAC use
> > case).
> > > > > > >
> > > > > > > The sharing endpoint via ExternalConsumer is the only layer of
> > > > > > indirection
> > > > > > > at the API entry-point, but then once the set of
> > > activePrincipalRoles
> > > > > is
> > > > > > > resolved, the underlying authz engine doesn't behave any
> > > differently
> > > > > from
> > > > > > > normal Polaris authz.
> > > > > > >
> > > > > > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <fl...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Thanks everyone for the discussion!
> > > > > > > >
> > > > > > > > As a next step discussed in today's community sync, we will
> > start
> > > > > > reviewing
> > > > > > > > the spec change.
> > > > > > > >
> > > > > > > > Hi Dennis, is this PR ready for review?
> > > > > > > > https://github.com/apache/polaris/pull/5446
> > > > > > > >
> > > > > > > > Yufei
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <
> > > > > travismbowen@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for this discussion Dennis!
> > > > > > > > >
> > > > > > > > > Wanted to weigh in with my thoughts and questions.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Do you prefer /api/shares-management/v1/
> > > > > > > > > > instead of adding to /api/management/v1?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > While I can see the argument for a share specific
> management
> > > > > endpoint
> > > > > > > > there
> > > > > > > > > doesn't seem to be a clear obvious choice from what I can
> see
> > > - I
> > > > > > lean
> > > > > > > > > towards /api/management as being the central place for all
> > the
> > > > > > > > management -
> > > > > > > > > share or not. As long as we're careful not to leak share
> > types
> > > or
> > > > > non
> > > > > > > > share
> > > > > > > > > types into the wrong api (which could be helped by having
> > > > separate
> > > > > > yamls)
> > > > > > > > > then it feels reasonable to have all under the management
> > api.
> > > > > > > > >
> > > > > > > > > Also went through the doc and PR and made several comments
> as
> > > > well.
> > > > > > > > >
> > > > > > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi All,
> > > > > > > > > >
> > > > > > > > > > Prithvi mentioned ExternalConsumer and principals. I'd
> like
> > > to
> > > > > > > > > > reinforce the need to further discuss "consumers" and
> > > > principals
> > > > > > and
> > > > > > > > how
> > > > > > > > > > they relate to IdP and Polaris Authentication.
> > > > > > > > > >
> > > > > > > > > > From my POV, share access control should not be a new
> > > mechanism
> > > > > but
> > > > > > > > > should
> > > > > > > > > > build on top of the existing extensible authentication /
> > > > > > authorization
> > > > > > > > > > framework. Supporting external IdP for shares would be a
> > > > valuable
> > > > > > > > feature
> > > > > > > > > > IMHO.
> > > > > > > > > >
> > > > > > > > > > Side note: I'm posting comments on PR 5446 too.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <
> > > > > > prithvisivasankar@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > Thank you for reviving this, and for the concrete spec
> in
> > > PR
> > > > > > 5446..
> > > > > > > > the
> > > > > > > > > > > journeys doc was especially helpful for seeing the
> > > end-to-end
> > > > > > flow.
> > > > > > > > > > >
> > > > > > > > > > > I agree this is worth doing, and that v1 is more than a
> > > > > tutorial
> > > > > > on
> > > > > > > > > > today's
> > > > > > > > > > > RBAC. A partner principal on /api/catalog is already
> > > > possible;
> > > > > > what
> > > > > > > > > seems
> > > > > > > > > > > valuable is a first-class share (enumerated members, a
> > > > > restricted
> > > > > > > > > > consumer,
> > > > > > > > > > > a listing as the binding and audit unit) plus a
> > segregated,
> > > > > > read-only
> > > > > > > > > > > Iceberg REST surface with credential vending. Stock
> > Iceberg
> > > > > REST
> > > > > > as
> > > > > > > > the
> > > > > > > > > > > consumer protocol, and keeping Flight and
> > > Polaris-to-Polaris
> > > > > > > > federation
> > > > > > > > > > out
> > > > > > > > > > > of v1, all sound right to me.
> > > > > > > > > > >
> > > > > > > > > > > I don't have a strong view yet on the URL layout
> > > (management
> > > > vs
> > > > > > > > shares
> > > > > > > > > > > prefix, nested vs top-level, prefix keyed by share vs
> > > > listing)
> > > > > > and am
> > > > > > > > > > happy
> > > > > > > > > > > to follow whatever the community prefers there.
> > > > > > > > > > >
> > > > > > > > > > > A few things I wanted to check on the spec itself,
> > several
> > > of
> > > > > > these
> > > > > > > > > echo
> > > > > > > > > > > points already raised on the list:
> > > > > > > > > > >
> > > > > > > > > > > 1. Membership as source of truth. If a hidden catalog
> > role
> > > is
> > > > > > > > > > independently
> > > > > > > > > > > grantable, a share could be widened without going
> through
> > > the
> > > > > > share
> > > > > > > > > APIs.
> > > > > > > > > > > Would it make sense to say the data plane authorizes
> > > against
> > > > > > > > > membership +
> > > > > > > > > > > listing, and that any backing roles are not
> user-visible
> > /
> > > > not
> > > > > > > > > > > independently grantable?
> > > > > > > > > > > 2. ExternalConsumer and principals. The Aug 7 recap
> > already
> > > > > > noted we
> > > > > > > > > > should
> > > > > > > > > > > not assume a PrincipalEntity behind every
> > ExternalConsumer.
> > > > The
> > > > > > > > > journeys
> > > > > > > > > > > still describe granting the consumer a share principal
> > > role,
> > > > I
> > > > > > assume
> > > > > > > > > > that
> > > > > > > > > > > is an implementation sketch for v1 client_id/secret,
> not
> > > the
> > > > > > public
> > > > > > > > > > model?
> > > > > > > > > > > 3. Listing expiration. JB's original note was "share
> > these
> > > > > tables
> > > > > > > > with
> > > > > > > > > > this
> > > > > > > > > > > consumer until this date." Credential expiry is a bit
> > > > > different.
> > > > > > > > Would
> > > > > > > > > a
> > > > > > > > > > > listing-level expiration (and maybe ACTIVE/SUSPENDED)
> be
> > in
> > > > > > scope for
> > > > > > > > > v1?
> > > > > > > > > > > 4. Time-travel. As raised in the community sync,
> sharing
> > > via
> > > > > > Iceberg
> > > > > > > > > REST
> > > > > > > > > > > means historical snapshots remain readable. Totally
> fine
> > > for
> > > > v1
> > > > > > to
> > > > > > > > > > support
> > > > > > > > > > > only full history, would an explicit field on the share
> > > help
> > > > > so a
> > > > > > > > later
> > > > > > > > > > > CURRENT_ONLY option is not a breaking surprise?
> > > > > > > > > > > 5. Consumer data plane. The current file is
> control-plane
> > > > only,
> > > > > > which
> > > > > > > > > is
> > > > > > > > > > a
> > > > > > > > > > > good first step. It might help to list the Iceberg
> routes
> > > > that
> > > > > > are
> > > > > > > > > > mounted,
> > > > > > > > > > > that writes 404, that non-members 404, and that /config
> > > does
> > > > > not
> > > > > > > > return
> > > > > > > > > > > provider catalog properties, so implementations don't
> > > > diverge.
> > > > > > > > > > >
> > > > > > > > > > > Two smaller notes, if useful: in the J1 journey the
> > > consumer
> > > > > > still
> > > > > > > > > posts
> > > > > > > > > > to
> > > > > > > > > > > /api/catalog/v1/oauth/tokens, if partners are not
> > supposed
> > > to
> > > > > > touch
> > > > > > > > the
> > > > > > > > > > > internal catalog API, should the default token endpoint
> > > live
> > > > on
> > > > > > the
> > > > > > > > > > shares
> > > > > > > > > > > surface (or an external IdP)? And on Dmitri's point
> about
> > > > share
> > > > > > > > > operators
> > > > > > > > > > > vs catalog operators, a SHARE_MANAGE privilege on the
> > bound
> > > > > > catalog
> > > > > > > > > > > (independent of CATALOG_MANAGE_ACCESS) would make that
> > work
> > > > on
> > > > > > any
> > > > > > > > URL
> > > > > > > > > > > prefix.
> > > > > > > > > > >
> > > > > > > > > > > Happy to review the next revision, and thanks again to
> > both
> > > > of
> > > > > > you
> > > > > > > > for
> > > > > > > > > > > driving this.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Prithvi S
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <
> > dhuo@apache.org
> > > >
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > As discussed live, looks like we both might have some
> > > > drafts
> > > > > > we can
> > > > > > > > > > > compare
> > > > > > > > > > > > and choose/merge from :)
> > > > > > > > > > > >
> > > > > > > > > > > > Here's my current draft spec PR; it opts for my
> > preferred
> > > > > > syntax
> > > > > > > > > > choices
> > > > > > > > > > > > for now (/api/management/ for share-management,
> > > > /api/shares/
> > > > > > for
> > > > > > > > > > consumer
> > > > > > > > > > > > data-plane):
> > https://github.com/apache/polaris/pull/5446
> > > > > > > > > > > >
> > > > > > > > > > > > Since we were talking about preferring mdfiles for
> > design
> > > > > > iteration
> > > > > > > > > > > here's
> > > > > > > > > > > > a companion design-addendum which just enumerates
> what
> > > the
> > > > > > > > end-to-end
> > > > > > > > > > > user
> > > > > > > > > > > > journeys look like syntactically for each combination
> > of
> > > > > design
> > > > > > > > > > choices:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > > > > > > > > > >
> > > > > > > > > > > > Please let me know if anyone prefers a Google Doc
> > version
> > > > of
> > > > > > the
> > > > > > > > > > > > design-addendum for easier commenting, or if the
> > > > > > design-addendum
> > > > > > > > > itself
> > > > > > > > > > > > should also be modeled as a PR on some free-floating
> > > branch
> > > > > > just to
> > > > > > > > > > > capture
> > > > > > > > > > > > comments.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <
> > > > > > > > jb@nanthrax.net
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I am resuming work on this proposal.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding our recent discussion, I will move
> forward
> > > with
> > > > > > > > opening a
> > > > > > > > > > > > > draft PR for the /api/shares/v1 endpoint to help
> > > > illustrate
> > > > > > the
> > > > > > > > > > > > > design.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I will update this thread once the PR is ready.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > JB
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov
> <
> > > > > > > > > dimas@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Dennis,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the recap!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I was thinking that share management is probably
> > > > distinct
> > > > > > from
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > management. A user who already has a
> realm+catalog
> > > may
> > > > > > have an
> > > > > > > > > > option
> > > > > > > > > > > > to
> > > > > > > > > > > > > > manage shares, but not the catalog itself. Would
> it
> > > > make
> > > > > > sense
> > > > > > > > to
> > > > > > > > > > > > > separate
> > > > > > > > > > > > > > the share management API from the general Polaris
> > > > > > management
> > > > > > > > > API? I
> > > > > > > > > > > > mean
> > > > > > > > > > > > > > using a different prefix for URIs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For example: /api/shares/v1/
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Obviously clients should not make any assumptions
> > > about
> > > > > the
> > > > > > > > base
> > > > > > > > > > path
> > > > > > > > > > > > > > (/api/shares), which could be different in
> > different
> > > > > > > > deployments
> > > > > > > > > > > (e.g.
> > > > > > > > > > > > > > going through a gateway). Clients must be able to
> > > take
> > > > > the
> > > > > > base
> > > > > > > > > URI
> > > > > > > > > > > as
> > > > > > > > > > > > a
> > > > > > > > > > > > > > config option. Path segments after /v1/ will be
> > > > specified
> > > > > > by
> > > > > > > > > > Polaris
> > > > > > > > > > > > > (Open
> > > > > > > > > > > > > > API yaml).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <
> > > > > dhuo@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Just to recap some of the discussion from
> today's
> > > > > > community
> > > > > > > > > sync,
> > > > > > > > > > > > > Dmitri,
> > > > > > > > > > > > > > > Robert, and Alex helped raise a few
> > considerations
> > > to
> > > > > > roll
> > > > > > > > into
> > > > > > > > > > the
> > > > > > > > > > > > > next
> > > > > > > > > > > > > > > design/prototyping phase:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. Given the way sharing is packaged as a
> > separate
> > > > > > thing, it
> > > > > > > > > may
> > > > > > > > > > > not
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > obvious to users that historical snapshots are
> > > > equally
> > > > > > > > > available
> > > > > > > > > > to
> > > > > > > > > > > > > > > consumers as the latest snapshot (by virtue of
> > > > sharing
> > > > > > via
> > > > > > > > the
> > > > > > > > > > > > > standard IRC
> > > > > > > > > > > > > > > protocol) - we should make sure to document
> that
> > > fact
> > > > > > well,
> > > > > > > > > > > including
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > pitfall that deleting rows by updating a table
> > > > doesn't
> > > > > > > > > > necessarily
> > > > > > > > > > > > mean
> > > > > > > > > > > > > > > those rows become inaccessible to consumers
> > > > > > > > > > > > > > > 2. Make sure not to hard-code assumptions about
> > the
> > > > > > existence
> > > > > > > > > of
> > > > > > > > > > an
> > > > > > > > > > > > > > > internal PrincipalEntity backing an
> > > ExternalConsumer
> > > > -
> > > > > > even
> > > > > > > > if
> > > > > > > > > > the
> > > > > > > > > > > > > first
> > > > > > > > > > > > > > > MVP implementation uses a nested/hidden
> Principal
> > > of
> > > > > some
> > > > > > > > type
> > > > > > > > > > that
> > > > > > > > > > > > > holds a
> > > > > > > > > > > > > > > Polaris-managed client_id/client_secret, the
> > > > 3rd-party
> > > > > > IdP
> > > > > > > > use
> > > > > > > > > > > cases
> > > > > > > > > > > > > > > require "implicit" Principals resolved via the
> > > oauth
> > > > > > token.
> > > > > > > > In
> > > > > > > > > > such
> > > > > > > > > > > > > cases
> > > > > > > > > > > > > > > the ExternalConsumer contains the metadata
> > > > coordinating
> > > > > > the
> > > > > > > > > > > three-way
> > > > > > > > > > > > > > > handshake between the data-provider,
> > data-consumer,
> > > > and
> > > > > > IdP,
> > > > > > > > > but
> > > > > > > > > > > > there
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > no PrincipalEntity underlying it.
> > > > > > > > > > > > > > > 3. Longer-term, we may want to tackle
> > > "on-behalf-of"
> > > > > > > > semantics
> > > > > > > > > -
> > > > > > > > > > > > > initially,
> > > > > > > > > > > > > > > the way an ExternalConsumer works is that the
> > > > > > finer-grained
> > > > > > > > > > > consumer
> > > > > > > > > > > > > users
> > > > > > > > > > > > > > > share a single ExternalConsumer identity, just
> > like
> > > > > how a
> > > > > > > > > > FEDERATED
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > shares a single service principal in its
> > > > > > ConnectionConfig.
> > > > > > > > But
> > > > > > > > > in
> > > > > > > > > > > > some
> > > > > > > > > > > > > > > cases the consumer may want some end-to-end
> > > > attribution
> > > > > > of
> > > > > > > > > > > > fine-grained
> > > > > > > > > > > > > > > users, like "on-behalf-of: bob @
> > > consumercompany.com
> > > > "
> > > > > > when
> > > > > > > > the
> > > > > > > > > > > > > > > ExternalConsumer implicit principal is
> something
> > > like
> > > > > > > > > > > > > > > "externalconsumer$consumercompany"
> > > > > > > > > > > > > > > 4. We can iterate more rapidly on a draft spec
> PR
> > > and
> > > > > > also
> > > > > > > > > expand
> > > > > > > > > > > on
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > detailed technical design
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <
> > > > > > dhuo@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks JB for reviving this thread, and
> belated
> > > > > thanks
> > > > > > to
> > > > > > > > all
> > > > > > > > > > > > > reviewers
> > > > > > > > > > > > > > > > who have left feedback in the doc or this
> > thread
> > > so
> > > > > > far!
> > > > > > > > And
> > > > > > > > > > > > > apologies
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the silence on this, it's still been on my
> > radar
> > > > but
> > > > > I
> > > > > > got
> > > > > > > > > > > > > distracted by
> > > > > > > > > > > > > > > > some other work for awhile.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Echoing JB's summary, the proposal is indeed
> > that
> > > > > we're
> > > > > > > > > mostly
> > > > > > > > > > > just
> > > > > > > > > > > > > > > > reusing existing Authz/RBAC machinery, and
> the
> > > > > > translation
> > > > > > > > > > layer
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > sharing concepts sits *on top of* the
> existing
> > > > authz
> > > > > > engine
> > > > > > > > > as
> > > > > > > > > > > much
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > So even if we're adding layers of indirection
> > on
> > > > > > > > > > > > > ExternalConsumer/Listing
> > > > > > > > > > > > > > > > ->
> Principal/PrincipalRole/CatalogRole+grants,
> > > the
> > > > > > parts
> > > > > > > > that
> > > > > > > > > > end
> > > > > > > > > > > > up
> > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > the underlying authz just reuse existing
> > > Authorizer
> > > > > > logic.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Based on some other offline discussion with
> > > > Dmitri, I
> > > > > > > > > realized
> > > > > > > > > > > why
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > discussion around "requirements" may have
> been
> > > > > > confusing:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. My attempts to shorten the doc
> inadvertently
> > > cut
> > > > > out
> > > > > > > > some
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > "abstract requirements" formulations
> > > > > > > > > > > > > > > > 2. The project in itself is somewhat more of
> a
> > > > "form
> > > > > > *as*
> > > > > > > > > > > function"
> > > > > > > > > > > > > > > > feature than *strictly* functional in nature
> --
> > > for
> > > > > > > > example,
> > > > > > > > > > many
> > > > > > > > > > > > > > > correctly
> > > > > > > > > > > > > > > > pointed out that "sharing" via the existing
> > > > > > RBAC/Principals
> > > > > > > > > > > syntax
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > already *possible* just by letting a
> cross-org
> > > > > > "consumer"
> > > > > > > > be
> > > > > > > > > a
> > > > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > principal and access your own company's main
> > > > Polaris
> > > > > > > > catalog
> > > > > > > > > > > > > endpoints.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For (1) I added this to the Requirements
> > section
> > > in
> > > > > > the doc
> > > > > > > > > to
> > > > > > > > > > > > better
> > > > > > > > > > > > > > > > clarify the abstract form of the data model
> > > > element:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *A. Sharing Container*: Container concept for
> > > > holding
> > > > > > > > things
> > > > > > > > > > you
> > > > > > > > > > > > > want to
> > > > > > > > > > > > > > > > share - “ShareEntity”
> > > > > > > > > > > > > > > > *B. Consumer Identity*: Special flavor of
> > > > > > principal-like
> > > > > > > > > entity
> > > > > > > > > > > > > > > > representing a consumer relationship -
> > > > > > “ExternalConsumer”
> > > > > > > > > > > > > > > > *C. Map Containers to Consumers*: A way to
> > > specify
> > > > > > letting
> > > > > > > > a
> > > > > > > > > > > given
> > > > > > > > > > > > > > > > consumer access a given sharing container -
> > > > “Listing”
> > > > > > > > > > > > > > > > *D. Consumer Endpoint*: A segregated API
> > > > access-point
> > > > > > for
> > > > > > > > > > > > consumers -
> > > > > > > > > > > > > > > > “EndpointConfig”
> > > > > > > > > > > > > > > > *E. Objects in Containers*: A way to specify
> > > which
> > > > > > > > in-catalog
> > > > > > > > > > > > objects
> > > > > > > > > > > > > > > > belong in the big sharing container -
> > > > > “ShareMembership”
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This still bakes in a bit of implementation
> > > detail
> > > > > > since it
> > > > > > > > > > begs
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > question why not just let core RBAC/grants
> > handle
> > > > (C)
> > > > > > and
> > > > > > > > > (E).
> > > > > > > > > > To
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > end,
> > > > > > > > > > > > > > > > and addressing problem (2) of where the "form
> > > *is*
> > > > > > > > > function", I
> > > > > > > > > > > > > added a
> > > > > > > > > > > > > > > > "Concept Mapping" tab:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think the details of more advanced features
> > can
> > > > > > still be
> > > > > > > > > > hashed
> > > > > > > > > > > > out
> > > > > > > > > > > > > > > > incrementally without blocking work on an MVP
> > > (and
> > > > > > > > certainly
> > > > > > > > > > the
> > > > > > > > > > > > MVP
> > > > > > > > > > > > > > > scope
> > > > > > > > > > > > > > > > could be trimmed so we can
> launch-and-iterate),
> > > so
> > > > as
> > > > > > long
> > > > > > > > as
> > > > > > > > > > > > there's
> > > > > > > > > > > > > > > > community alignment on the general direction
> I
> > > > > propose
> > > > > > we
> > > > > > > > > could
> > > > > > > > > > > > move
> > > > > > > > > > > > > > > > forward on basic building blocks. Or if
> > building
> > > > > blocks
> > > > > > > > > > > themselves
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > controversial we could still proceed with
> some
> > > > > > prototyping
> > > > > > > > > > across
> > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > options.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM
> Jean-Baptiste
> > > > > Onofré <
> > > > > > > > > > > > > jb@nanthrax.net>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> Hi everyone,
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> I would like to resume the discussion about
> > Data
> > > > > > Sharing
> > > > > > > > in
> > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > >> and propose focusing on three major items to
> > > help
> > > > us
> > > > > > move
> > > > > > > > > > > forward.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> 1. What is data sharing?
> > > > > > > > > > > > > > > >> In a nutshell, it is an open protocol
> > > proposal
> > > > > > > > allowing
> > > > > > > > > > > > > > > >> organizations to share live data without
> > copying
> > > > or
> > > > > > > > > > replication.
> > > > > > > > > > > > The
> > > > > > > > > > > > > > > >> main goal is zero-copy sharing achieved
> > through
> > > > > shared
> > > > > > > > > > metadata
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > >> authorization.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> On Polaris, this would mean:
> > > > > > > > > > > > > > > >> a. The consumer is authenticated as usual.
> > > > > > > > > > > > > > > >> b. The Polaris server provides the consumer
> > with
> > > > > > direct
> > > > > > > > > access
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> data files (e.g., Parquet, as specified in
> the
> > > > > > > > > metadata.json).
> > > > > > > > > > > > > > > >> c. The consumer reads the data files
> directly
> > > > using
> > > > > > their
> > > > > > > > > > > > preferred
> > > > > > > > > > > > > > > >> tool (e.g., a query engine).
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> While Polaris already has most of the
> > necessary
> > > > > > underlying
> > > > > > > > > > > > > > > >> capabilities, we need clear
> branding/packaging
> > > > > around
> > > > > > this
> > > > > > > > > use
> > > > > > > > > > > > case.
> > > > > > > > > > > > > > > >> Specifically, this means:
> > > > > > > > > > > > > > > >> a. A share is a dedicated catalog in the
> > Polaris
> > > > > > server,
> > > > > > > > or
> > > > > > > > > a
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > >> role scoped to specific namespaces.
> > > > > > > > > > > > > > > >> b. A recipient is represented by a principal
> > > and a
> > > > > > > > principal
> > > > > > > > > > > role.
> > > > > > > > > > > > > > > >> c. A consumer profile consists of the
> catalog
> > > URI
> > > > > and
> > > > > > the
> > > > > > > > > > OAuth2
> > > > > > > > > > > > > > > >> client_id/client_secret.
> > > > > > > > > > > > > > > >> d. We use credential vending (temporary STS
> > > > > > credentials)
> > > > > > > > for
> > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > >> access to the data files.
> > > > > > > > > > > > > > > >> e. We use the Polaris IRC layer to expose
> the
> > > > > > metadata and
> > > > > > > > > the
> > > > > > > > > > > > file
> > > > > > > > > > > > > > > >> locations/access.
> > > > > > > > > > > > > > > >> f. We can also support Polaris-to-Polaris
> > > sharing
> > > > > via
> > > > > > > > > > > Federation.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Concretely:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> - On the provider side, we use the
> standard
> > > > > Polaris
> > > > > > RBAC
> > > > > > > > > > model
> > > > > > > > > > > > > > > >> (assigning grants to catalog roles, and
> > > assigning
> > > > > > those
> > > > > > > > > > catalog
> > > > > > > > > > > > > roles
> > > > > > > > > > > > > > > >> to principal roles):
> > > > > > > > > > > > > > > >> - Create a dedicated principal with
> its
> > > own
> > > > > > > > > > > > > > > client_id/client_secret.
> > > > > > > > > > > > > > > >> - Create a catalog role with
> > > > TABLE_READ_DATA,
> > > > > > > > > > TABLE_LIST,
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > >> NAMESPACE_LIST grants on the namespaces to
> be
> > > > shared
> > > > > > (no
> > > > > > > > > write
> > > > > > > > > > > > > > > >> grants).
> > > > > > > > > > > > > > > >> - Create a principal role per
> > destination,
> > > > > > attach
> > > > > > > > the
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > >> role, and assign the principal.
> > > > > > > > > > > > > > > >> - The destination engine uses
> loadTable
> > > with
> > > > > the
> > > > > > > > > > > > > > > >> "X-Iceberg-Access-Delegation:
> > > vended-credentials"
> > > > > > header.
> > > > > > > > > > > Polaris
> > > > > > > > > > > > > then
> > > > > > > > > > > > > > > >> emits temporary, read-only storage
> > credentials,
> > > > > > allowing
> > > > > > > > the
> > > > > > > > > > > > engine
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > >> read the data files directly from the
> owner's
> > > > bucket
> > > > > > > > without
> > > > > > > > > > > > > accessing
> > > > > > > > > > > > > > > >> the rest of the bucket. This achieves
> > zero-copy
> > > > > > sharing.
> > > > > > > > > > > > > > > >> - On the recipient/consumer side, no
> special
> > > > > > > > configuration
> > > > > > > > > > is
> > > > > > > > > > > > > > > >> needed; any IRC client can use it.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> 2. What is data sharing not?
> > > > > > > > > > > > > > > >> I previously considered adding an Arrow
> > > Flight
> > > > > > > > endpoint
> > > > > > > > > to
> > > > > > > > > > > > > > > >> Polaris, but I no longer think it is a good
> > idea
> > > > for
> > > > > > the
> > > > > > > > > > initial
> > > > > > > > > > > > > > > >> phase. An Arrow Flight endpoint would serve
> > the
> > > > data
> > > > > > > > > directly,
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > >> would require Polaris to read the data and
> > > expose
> > > > > it.
> > > > > > I am
> > > > > > > > > > > against
> > > > > > > > > > > > > > > >> embedding a query engine in Polaris, as this
> > > tight
> > > > > > > > coupling
> > > > > > > > > > > feels
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > > >> an anti-pattern. While we could use a
> > delegation
> > > > > > service
> > > > > > > > to
> > > > > > > > > > > serve
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> Flight endpoint, I think it should be
> > considered
> > > > > for a
> > > > > > > > later
> > > > > > > > > > > > phase,
> > > > > > > > > > > > > if
> > > > > > > > > > > > > > > >> at all, since a Flight endpoint might be
> > better
> > > > > > suited as
> > > > > > > > a
> > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > >> responsibility and could encourage data
> > copying.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> The first phase should focus on packaging
> and
> > > > > > branding the
> > > > > > > > > > > > > > > >> capabilities we already have.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> 3. What could the Data Sharing "package"
> look
> > > > like
> > > > > in
> > > > > > > > > > Polaris?
> > > > > > > > > > > > > > > >> To make Data Sharing obvious and easy to
> > > use,
> > > > we
> > > > > > > > should
> > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > >> the missing concept of a "share"—a named
> > > > definition
> > > > > > that
> > > > > > > > > users
> > > > > > > > > > > can
> > > > > > > > > > > > > > > >> list, discover, and define. The goal is to
> > > > > > materialize the
> > > > > > > > > > > user's
> > > > > > > > > > > > > > > >> intent ("I want to share these tables with
> > this
> > > > > > consumer
> > > > > > > > > until
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > >> date") into standard Polaris entities
> > > (principal,
> > > > > > > > principal
> > > > > > > > > > > role,
> > > > > > > > > > > > > > > >> catalog role, and grants).
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Therefore, I propose adding a /shares
> endpoint
> > > to
> > > > > the
> > > > > > > > > Polaris
> > > > > > > > > > > > > > > >> Management API to package and materialize
> > these
> > > > > share
> > > > > > > > > > > definitions
> > > > > > > > > > > > > into
> > > > > > > > > > > > > > > >> Polaris entities. This would provide a
> usable,
> > > > > > > > first-version
> > > > > > > > > > > > package
> > > > > > > > > > > > > > > >> for Polaris Data Sharing.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> I hope this addresses Dmitri's questions and
> > > > > provides
> > > > > > a
> > > > > > > > more
> > > > > > > > > > > > > concrete
> > > > > > > > > > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Thoughts?
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Regards,
> > > > > > > > > > > > > > > >> JB
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <
> > > > > > > > dhuo@apache.org
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > Hi All,
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > One big emerging enterprise use case
> coming
> > up
> > > > as
> > > > > > more
> > > > > > > > > > people
> > > > > > > > > > > > > > > >> consolidate
> > > > > > > > > > > > > > > >> > Data Lakehouses and Catalogs is something
> > > > commonly
> > > > > > known
> > > > > > > > > as
> > > > > > > > > > > > "Data
> > > > > > > > > > > > > > > >> Sharing",
> > > > > > > > > > > > > > > >> > an more specifically over the course of
> > > adoption
> > > > > of
> > > > > > open
> > > > > > > > > > table
> > > > > > > > > > > > > formats
> > > > > > > > > > > > > > > >> > "Open Data Sharing".
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > Examples of existing managed service
> > > providers'
> > > > > Data
> > > > > > > > > Sharing
> > > > > > > > > > > > > features:
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> >
> > > > https://www.databricks.com/product/delta-sharing
> > > > > > > > > > > > > > > >> >
> > > > > > > > >
> https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > The basic idea is that when you share data
> > > > between
> > > > > > > > > different
> > > > > > > > > > > > > > > companies,
> > > > > > > > > > > > > > > >> you
> > > > > > > > > > > > > > > >> > need a first-class governance/management
> > layer
> > > > and
> > > > > > extra
> > > > > > > > > > > > > > > >> bells-and-whistles
> > > > > > > > > > > > > > > >> > that are distinct from just the basic
> > > > capabilities
> > > > > > of
> > > > > > > > RBAC
> > > > > > > > > > or
> > > > > > > > > > > > > > > >> generalized
> > > > > > > > > > > > > > > >> > access-control (i.e. if you're sharing
> > across
> > > > > > > > > > > > partially-untrusted
> > > > > > > > > > > > > org
> > > > > > > > > > > > > > > >> > boundaries, you don't just let the
> consumer
> > > > > > organization
> > > > > > > > > log
> > > > > > > > > > > > into
> > > > > > > > > > > > > your
> > > > > > > > > > > > > > > >> > datalake like one of your own employees).
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > JB and I put together this high-level
> > proposal
> > > > for
> > > > > > > > > > supporting
> > > > > > > > > > > > Open
> > > > > > > > > > > > > > > >> Sharing
> > > > > > > > > > > > > > > >> > in Polaris:
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > Tentatively, it means adding ~5 logical
> data
> > > > model
> > > > > > > > > > constructs,
> > > > > > > > > > > > > some of
> > > > > > > > > > > > > > > >> > which may be a first-class PolarisEntity
> > type,
> > > > > > others
> > > > > > > > > > subtypes
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > >> existing
> > > > > > > > > > > > > > > >> > entities, and others just a nested
> > construct:
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > - ShareEntity (would behave similarly
> to
> > a
> > > > > > Catalog)
> > > > > > > > > > > > > > > >> > - ExternalConsumer (mostly inherits
> from
> > > > > > Principal)
> > > > > > > > > > > > > > > >> > - Listing (Similar to a "role grant"
> but
> > > has
> > > > > > > > different
> > > > > > > > > > > > > metadata)
> > > > > > > > > > > > > > > >> > - EndpointConfig (nested config under
> > > > Listing)
> > > > > > > > > > > > > > > >> > - ShareMembership (Similar to a
> > "securable
> > > > > > grant" but
> > > > > > > > > > > > different
> > > > > > > > > > > > > > > >> metadata)
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > Feedback/comments welcome! I'll also bring
> > it
> > > up
> > > > > for
> > > > > > > > live
> > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > >> > there's time in the community sync.
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > Cheers,
> > > > > > > > > > > > > > > >> > Dennis
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Dennis,
Let me try to restate my concern from another angle. Apologies that I was
not very clear in my previous messages.
Currently the create listing doc says: "Create a listing, which grants an
external consumer access to this share".
So, this binds authorization decisions about a Share (Listing) to
ExternalConsumer.
In IRC use cases a PolarisPrincipal is established by authenticating the
client request (either via internal JWT or via external IdP), Subsequently,
the PolarisPrincipal is passed to PolarisAuthorizer for making
authorization decisions. The latter part can be external too (e.g. OPA /
Ranger).
(Side note: the "external" work in ExternalConsumer is actually quite
confusing with respect to prior usage of that word in external IdP use
cases.)
In my view, the ExternalConsumer and related concepts in current Shares API
do not align with the external IdP and Authorizer workflows.
IIRC, actual access to shared tables is going to use the IRC API, right?
That access path in existing Polaris use cases maps well to external IdP
and external authorizers.
Why does the Shares API need to define a "Consumer" as a separate entity
and narrow authorization decisions to its relationship with Listings?
I believe authentication and authorization options should remain as wide as
possible because these matters are usually addressed very differently in
each deployment.
To be clear: I do not mind defining Share "Consumers" in the internal
Polaris RBAC domain. It can fit well there, I'm only suggesting keeping the
Shares API itself neutral to Auth N/Z concerns.
Cheers,
Dmitri.
On Wed, Oct 21, 2026 at 7:02 PM Dennis Huo <dh...@apache.org> wrote:
> Ah thanks for clarifying - to be clear, the ConsumerCredential is *not*
> strictly necessary for the Listing in terms of spec expressibility, only
> the ExternalConsumer is.
>
> An ExternalConsumer doesn't need to be backed by a ConsumerCredential; the
> ExternalConsumer is an abstract Sharing-specific pointer to whatever does
> the indirection in the sharing implementation.
>
> Responded to your github comment - I'll split anything that touches
> "ConsumerCredential" into a separate yaml file so it's easier to see the
> core auth-agnostic sharing spec in one file, with the more
> implementation-specific decorations around ConsumerCredential in an
> auxiliary file.
>
> I suppose another option is to even split it into separate PRs (in addition
> to separate yaml files?). Then we can easily see one commit that's just the
> pure abstract auth-agnostic API, that in theory private Polaris
> implementations could already take and implement on external-IdP
> architectures to support the same agnostic spec.
>
> On Wed, Oct 21, 2026 at 1:54 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Dennis,
> >
> > While I still support the idea of splitting OpenAPI yaml files and URI
> > bases between the Management and Sharing APIs, I do not think it is
> tightly
> > coupled with my proposal for the authentication-neural Sharing API spec.
> >
> > To give an example, currently [5446] CreateListingRequest includes a
> > "consumerName". This immediately binds the Listing definition to
> > an ExternalConsumer and by extension to ConsumerCredential.
> >
> > ConsumerCredential is naturally part of the authentication process during
> > Share access. I believe ConsumerCredential (and related entities) is
> worth
> > isolating from the Shares API to increase interoperability.
> >
> > Implementations that choose to rely on ConsumerCredential for Share
> access
> > can still do so. However, ConsumerCredential will not be a requirement
> for
> > all possible Share API implementations.
> >
> > This can work with any OpenAPI yaml and URI paths layout.
> >
> > I'd like to resolve this aspect before we go back to URI paths and YAML
> > file locations. Actually, I do not feel too strongly about path
> prefixes. I
> > do believe separate prefixes make implementations easier in many aspects
> > (including network setup), but interleaving those paths with the
> Management
> > API is not a blocker, from my POV, only an inconvenience.
> >
> > Cheers,
> > Dmitri.
> >
> > [5446] https://github.com/apache/polaris/pull/5446
> >
> > On Wed, Oct 21, 2026 at 3:19 PM Dennis Huo <dh...@apache.org> wrote:
> >
> > > Yeah, that's a good topic to discuss; I guess it touches on our core
> > > (unresolved?) conventions on how we organize API url-prefixes and yaml
> > > files. Reposting my response to the github comment as well:
> > >
> > > Right, I think we could certainly split the API routes and
> > yaml-definitions
> > > even further if we want to ensure a well-defined auth-detail-free core,
> > but
> > > for the standalone-Polaris use case we'd still need to define it
> > > *somewhere*
> > > under a Sharing-specific surface, right?
> > >
> > > @dimas-b <https://github.com/dimas-b> Were you thinking of just
> > splitting
> > > the yaml file but keeping the share-management root, or adding a third
> > > sharing prefix like
> > >
> > >
> >
> /api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials...
> > > ?
> > >
> > > Or just yaml-file split would be keeping
> > > /api/shares-management/v1/external-consumers/{name}/credentials... but
> > > putting it in a separate yaml file to make it easy for service-runners
> to
> > > choose not to support that whole yaml file's concrete APIs.
> > >
> > > On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Re: the shating API PR [5446], I'm repeating my GH comment here for
> > > > visibility.
> > > >
> > > > I'd like the API to be neutral from the authentication perspective.
> > > >
> > > > This is how we handle the IRC spec in Polaris. The API declares that
> > > > clients need to authenticate, but specific details about principals,
> > > > clients and secrets are delegated to the server.
> > > >
> > > > I think this is a sound approach for the sharing API too. Specific
> > > details
> > > > related to native Polaris principals (ConsumerCredential, etc.) can
> > > > probably remain in the Management API.
> > > >
> > > > WDYT?
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > [5446] https://github.com/apache/polaris/pull/5446
> > > >
> > > > On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > As I said during the community meeting, let's focus on the API PR
> > > > > first. We will build from there.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <dh...@apache.org> wrote:
> > > > > >
> > > > > > +1, thanks for the discussion everyone!
> > > > > >
> > > > > > Yes, I meant to flip it to "ready to review" earlier; flipped it
> > now.
> > > > I'm
> > > > > > still working through the existing set of comments and may need
> to
> > > come
> > > > > > back to it in the next few days before getting proper replies,
> but
> > > feel
> > > > > > free to add additional comments.
> > > > > >
> > > > > > Recapping the community sync discussion re: Dmitri's last
> message,
> > > I'm
> > > > > > fully aligned on building the sharing on top of existing authz
> > > > > mechanisms,
> > > > > > instead of being a new mechanism, and indeed external IdP is
> > intended
> > > > to
> > > > > be
> > > > > > a first-class supported use case.
> > > > > >
> > > > > > The latest distinction is that the ExternalConsumer isn't
> > > fundamentally
> > > > > > identified by any internal PolarisPrincipal, but rather that it
> > > serves
> > > > > as a
> > > > > > wrapper that references some identity, whether that identity is
> > > defined
> > > > > in
> > > > > > an external IdP without any latent in-Polaris representation or
> if
> > > > it's a
> > > > > > hidden PolarisPrincipal (for the Polaris-standalone RBAC use
> case).
> > > > > >
> > > > > > The sharing endpoint via ExternalConsumer is the only layer of
> > > > > indirection
> > > > > > at the API entry-point, but then once the set of
> > activePrincipalRoles
> > > > is
> > > > > > resolved, the underlying authz engine doesn't behave any
> > differently
> > > > from
> > > > > > normal Polaris authz.
> > > > > >
> > > > > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <fl...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Thanks everyone for the discussion!
> > > > > > >
> > > > > > > As a next step discussed in today's community sync, we will
> start
> > > > > reviewing
> > > > > > > the spec change.
> > > > > > >
> > > > > > > Hi Dennis, is this PR ready for review?
> > > > > > > https://github.com/apache/polaris/pull/5446
> > > > > > >
> > > > > > > Yufei
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <
> > > > travismbowen@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for this discussion Dennis!
> > > > > > > >
> > > > > > > > Wanted to weigh in with my thoughts and questions.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Do you prefer /api/shares-management/v1/
> > > > > > > > > instead of adding to /api/management/v1?
> > > > > > > >
> > > > > > > >
> > > > > > > > While I can see the argument for a share specific management
> > > > endpoint
> > > > > > > there
> > > > > > > > doesn't seem to be a clear obvious choice from what I can see
> > - I
> > > > > lean
> > > > > > > > towards /api/management as being the central place for all
> the
> > > > > > > management -
> > > > > > > > share or not. As long as we're careful not to leak share
> types
> > or
> > > > non
> > > > > > > share
> > > > > > > > types into the wrong api (which could be helped by having
> > > separate
> > > > > yamls)
> > > > > > > > then it feels reasonable to have all under the management
> api.
> > > > > > > >
> > > > > > > > Also went through the doc and PR and made several comments as
> > > well.
> > > > > > > >
> > > > > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <
> > > > > dimas@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > Prithvi mentioned ExternalConsumer and principals. I'd like
> > to
> > > > > > > > > reinforce the need to further discuss "consumers" and
> > > principals
> > > > > and
> > > > > > > how
> > > > > > > > > they relate to IdP and Polaris Authentication.
> > > > > > > > >
> > > > > > > > > From my POV, share access control should not be a new
> > mechanism
> > > > but
> > > > > > > > should
> > > > > > > > > build on top of the existing extensible authentication /
> > > > > authorization
> > > > > > > > > framework. Supporting external IdP for shares would be a
> > > valuable
> > > > > > > feature
> > > > > > > > > IMHO.
> > > > > > > > >
> > > > > > > > > Side note: I'm posting comments on PR 5446 too.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <
> > > > > prithvisivasankar@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi everyone,
> > > > > > > > > >
> > > > > > > > > > Thank you for reviving this, and for the concrete spec in
> > PR
> > > > > 5446..
> > > > > > > the
> > > > > > > > > > journeys doc was especially helpful for seeing the
> > end-to-end
> > > > > flow.
> > > > > > > > > >
> > > > > > > > > > I agree this is worth doing, and that v1 is more than a
> > > > tutorial
> > > > > on
> > > > > > > > > today's
> > > > > > > > > > RBAC. A partner principal on /api/catalog is already
> > > possible;
> > > > > what
> > > > > > > > seems
> > > > > > > > > > valuable is a first-class share (enumerated members, a
> > > > restricted
> > > > > > > > > consumer,
> > > > > > > > > > a listing as the binding and audit unit) plus a
> segregated,
> > > > > read-only
> > > > > > > > > > Iceberg REST surface with credential vending. Stock
> Iceberg
> > > > REST
> > > > > as
> > > > > > > the
> > > > > > > > > > consumer protocol, and keeping Flight and
> > Polaris-to-Polaris
> > > > > > > federation
> > > > > > > > > out
> > > > > > > > > > of v1, all sound right to me.
> > > > > > > > > >
> > > > > > > > > > I don't have a strong view yet on the URL layout
> > (management
> > > vs
> > > > > > > shares
> > > > > > > > > > prefix, nested vs top-level, prefix keyed by share vs
> > > listing)
> > > > > and am
> > > > > > > > > happy
> > > > > > > > > > to follow whatever the community prefers there.
> > > > > > > > > >
> > > > > > > > > > A few things I wanted to check on the spec itself,
> several
> > of
> > > > > these
> > > > > > > > echo
> > > > > > > > > > points already raised on the list:
> > > > > > > > > >
> > > > > > > > > > 1. Membership as source of truth. If a hidden catalog
> role
> > is
> > > > > > > > > independently
> > > > > > > > > > grantable, a share could be widened without going through
> > the
> > > > > share
> > > > > > > > APIs.
> > > > > > > > > > Would it make sense to say the data plane authorizes
> > against
> > > > > > > > membership +
> > > > > > > > > > listing, and that any backing roles are not user-visible
> /
> > > not
> > > > > > > > > > independently grantable?
> > > > > > > > > > 2. ExternalConsumer and principals. The Aug 7 recap
> already
> > > > > noted we
> > > > > > > > > should
> > > > > > > > > > not assume a PrincipalEntity behind every
> ExternalConsumer.
> > > The
> > > > > > > > journeys
> > > > > > > > > > still describe granting the consumer a share principal
> > role,
> > > I
> > > > > assume
> > > > > > > > > that
> > > > > > > > > > is an implementation sketch for v1 client_id/secret, not
> > the
> > > > > public
> > > > > > > > > model?
> > > > > > > > > > 3. Listing expiration. JB's original note was "share
> these
> > > > tables
> > > > > > > with
> > > > > > > > > this
> > > > > > > > > > consumer until this date." Credential expiry is a bit
> > > > different.
> > > > > > > Would
> > > > > > > > a
> > > > > > > > > > listing-level expiration (and maybe ACTIVE/SUSPENDED) be
> in
> > > > > scope for
> > > > > > > > v1?
> > > > > > > > > > 4. Time-travel. As raised in the community sync, sharing
> > via
> > > > > Iceberg
> > > > > > > > REST
> > > > > > > > > > means historical snapshots remain readable. Totally fine
> > for
> > > v1
> > > > > to
> > > > > > > > > support
> > > > > > > > > > only full history, would an explicit field on the share
> > help
> > > > so a
> > > > > > > later
> > > > > > > > > > CURRENT_ONLY option is not a breaking surprise?
> > > > > > > > > > 5. Consumer data plane. The current file is control-plane
> > > only,
> > > > > which
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > > good first step. It might help to list the Iceberg routes
> > > that
> > > > > are
> > > > > > > > > mounted,
> > > > > > > > > > that writes 404, that non-members 404, and that /config
> > does
> > > > not
> > > > > > > return
> > > > > > > > > > provider catalog properties, so implementations don't
> > > diverge.
> > > > > > > > > >
> > > > > > > > > > Two smaller notes, if useful: in the J1 journey the
> > consumer
> > > > > still
> > > > > > > > posts
> > > > > > > > > to
> > > > > > > > > > /api/catalog/v1/oauth/tokens, if partners are not
> supposed
> > to
> > > > > touch
> > > > > > > the
> > > > > > > > > > internal catalog API, should the default token endpoint
> > live
> > > on
> > > > > the
> > > > > > > > > shares
> > > > > > > > > > surface (or an external IdP)? And on Dmitri's point about
> > > share
> > > > > > > > operators
> > > > > > > > > > vs catalog operators, a SHARE_MANAGE privilege on the
> bound
> > > > > catalog
> > > > > > > > > > (independent of CATALOG_MANAGE_ACCESS) would make that
> work
> > > on
> > > > > any
> > > > > > > URL
> > > > > > > > > > prefix.
> > > > > > > > > >
> > > > > > > > > > Happy to review the next revision, and thanks again to
> both
> > > of
> > > > > you
> > > > > > > for
> > > > > > > > > > driving this.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Prithvi S
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <
> dhuo@apache.org
> > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > As discussed live, looks like we both might have some
> > > drafts
> > > > > we can
> > > > > > > > > > compare
> > > > > > > > > > > and choose/merge from :)
> > > > > > > > > > >
> > > > > > > > > > > Here's my current draft spec PR; it opts for my
> preferred
> > > > > syntax
> > > > > > > > > choices
> > > > > > > > > > > for now (/api/management/ for share-management,
> > > /api/shares/
> > > > > for
> > > > > > > > > consumer
> > > > > > > > > > > data-plane):
> https://github.com/apache/polaris/pull/5446
> > > > > > > > > > >
> > > > > > > > > > > Since we were talking about preferring mdfiles for
> design
> > > > > iteration
> > > > > > > > > > here's
> > > > > > > > > > > a companion design-addendum which just enumerates what
> > the
> > > > > > > end-to-end
> > > > > > > > > > user
> > > > > > > > > > > journeys look like syntactically for each combination
> of
> > > > design
> > > > > > > > > choices:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > > > > > > > > >
> > > > > > > > > > > Please let me know if anyone prefers a Google Doc
> version
> > > of
> > > > > the
> > > > > > > > > > > design-addendum for easier commenting, or if the
> > > > > design-addendum
> > > > > > > > itself
> > > > > > > > > > > should also be modeled as a PR on some free-floating
> > branch
> > > > > just to
> > > > > > > > > > capture
> > > > > > > > > > > comments.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <
> > > > > > > jb@nanthrax.net
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > I am resuming work on this proposal.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding our recent discussion, I will move forward
> > with
> > > > > > > opening a
> > > > > > > > > > > > draft PR for the /api/shares/v1 endpoint to help
> > > illustrate
> > > > > the
> > > > > > > > > > > > design.
> > > > > > > > > > > >
> > > > > > > > > > > > I will update this thread once the PR is ready.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > JB
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <
> > > > > > > > dimas@apache.org
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Dennis,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the recap!
> > > > > > > > > > > > >
> > > > > > > > > > > > > I was thinking that share management is probably
> > > distinct
> > > > > from
> > > > > > > > > > catalog
> > > > > > > > > > > > > management. A user who already has a realm+catalog
> > may
> > > > > have an
> > > > > > > > > option
> > > > > > > > > > > to
> > > > > > > > > > > > > manage shares, but not the catalog itself. Would it
> > > make
> > > > > sense
> > > > > > > to
> > > > > > > > > > > > separate
> > > > > > > > > > > > > the share management API from the general Polaris
> > > > > management
> > > > > > > > API? I
> > > > > > > > > > > mean
> > > > > > > > > > > > > using a different prefix for URIs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For example: /api/shares/v1/
> > > > > > > > > > > > >
> > > > > > > > > > > > > Obviously clients should not make any assumptions
> > about
> > > > the
> > > > > > > base
> > > > > > > > > path
> > > > > > > > > > > > > (/api/shares), which could be different in
> different
> > > > > > > deployments
> > > > > > > > > > (e.g.
> > > > > > > > > > > > > going through a gateway). Clients must be able to
> > take
> > > > the
> > > > > base
> > > > > > > > URI
> > > > > > > > > > as
> > > > > > > > > > > a
> > > > > > > > > > > > > config option. Path segments after /v1/ will be
> > > specified
> > > > > by
> > > > > > > > > Polaris
> > > > > > > > > > > > (Open
> > > > > > > > > > > > > API yaml).
> > > > > > > > > > > > >
> > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <
> > > > dhuo@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Just to recap some of the discussion from today's
> > > > > community
> > > > > > > > sync,
> > > > > > > > > > > > Dmitri,
> > > > > > > > > > > > > > Robert, and Alex helped raise a few
> considerations
> > to
> > > > > roll
> > > > > > > into
> > > > > > > > > the
> > > > > > > > > > > > next
> > > > > > > > > > > > > > design/prototyping phase:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Given the way sharing is packaged as a
> separate
> > > > > thing, it
> > > > > > > > may
> > > > > > > > > > not
> > > > > > > > > > > be
> > > > > > > > > > > > > > obvious to users that historical snapshots are
> > > equally
> > > > > > > > available
> > > > > > > > > to
> > > > > > > > > > > > > > consumers as the latest snapshot (by virtue of
> > > sharing
> > > > > via
> > > > > > > the
> > > > > > > > > > > > standard IRC
> > > > > > > > > > > > > > protocol) - we should make sure to document that
> > fact
> > > > > well,
> > > > > > > > > > including
> > > > > > > > > > > > the
> > > > > > > > > > > > > > pitfall that deleting rows by updating a table
> > > doesn't
> > > > > > > > > necessarily
> > > > > > > > > > > mean
> > > > > > > > > > > > > > those rows become inaccessible to consumers
> > > > > > > > > > > > > > 2. Make sure not to hard-code assumptions about
> the
> > > > > existence
> > > > > > > > of
> > > > > > > > > an
> > > > > > > > > > > > > > internal PrincipalEntity backing an
> > ExternalConsumer
> > > -
> > > > > even
> > > > > > > if
> > > > > > > > > the
> > > > > > > > > > > > first
> > > > > > > > > > > > > > MVP implementation uses a nested/hidden Principal
> > of
> > > > some
> > > > > > > type
> > > > > > > > > that
> > > > > > > > > > > > holds a
> > > > > > > > > > > > > > Polaris-managed client_id/client_secret, the
> > > 3rd-party
> > > > > IdP
> > > > > > > use
> > > > > > > > > > cases
> > > > > > > > > > > > > > require "implicit" Principals resolved via the
> > oauth
> > > > > token.
> > > > > > > In
> > > > > > > > > such
> > > > > > > > > > > > cases
> > > > > > > > > > > > > > the ExternalConsumer contains the metadata
> > > coordinating
> > > > > the
> > > > > > > > > > three-way
> > > > > > > > > > > > > > handshake between the data-provider,
> data-consumer,
> > > and
> > > > > IdP,
> > > > > > > > but
> > > > > > > > > > > there
> > > > > > > > > > > > is
> > > > > > > > > > > > > > no PrincipalEntity underlying it.
> > > > > > > > > > > > > > 3. Longer-term, we may want to tackle
> > "on-behalf-of"
> > > > > > > semantics
> > > > > > > > -
> > > > > > > > > > > > initially,
> > > > > > > > > > > > > > the way an ExternalConsumer works is that the
> > > > > finer-grained
> > > > > > > > > > consumer
> > > > > > > > > > > > users
> > > > > > > > > > > > > > share a single ExternalConsumer identity, just
> like
> > > > how a
> > > > > > > > > FEDERATED
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > shares a single service principal in its
> > > > > ConnectionConfig.
> > > > > > > But
> > > > > > > > in
> > > > > > > > > > > some
> > > > > > > > > > > > > > cases the consumer may want some end-to-end
> > > attribution
> > > > > of
> > > > > > > > > > > fine-grained
> > > > > > > > > > > > > > users, like "on-behalf-of: bob @
> > consumercompany.com
> > > "
> > > > > when
> > > > > > > the
> > > > > > > > > > > > > > ExternalConsumer implicit principal is something
> > like
> > > > > > > > > > > > > > "externalconsumer$consumercompany"
> > > > > > > > > > > > > > 4. We can iterate more rapidly on a draft spec PR
> > and
> > > > > also
> > > > > > > > expand
> > > > > > > > > > on
> > > > > > > > > > > > the
> > > > > > > > > > > > > > detailed technical design
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <
> > > > > dhuo@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks JB for reviving this thread, and belated
> > > > thanks
> > > > > to
> > > > > > > all
> > > > > > > > > > > > reviewers
> > > > > > > > > > > > > > > who have left feedback in the doc or this
> thread
> > so
> > > > > far!
> > > > > > > And
> > > > > > > > > > > > apologies
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the silence on this, it's still been on my
> radar
> > > but
> > > > I
> > > > > got
> > > > > > > > > > > > distracted by
> > > > > > > > > > > > > > > some other work for awhile.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Echoing JB's summary, the proposal is indeed
> that
> > > > we're
> > > > > > > > mostly
> > > > > > > > > > just
> > > > > > > > > > > > > > > reusing existing Authz/RBAC machinery, and the
> > > > > translation
> > > > > > > > > layer
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > sharing concepts sits *on top of* the existing
> > > authz
> > > > > engine
> > > > > > > > as
> > > > > > > > > > much
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > So even if we're adding layers of indirection
> on
> > > > > > > > > > > > ExternalConsumer/Listing
> > > > > > > > > > > > > > > -> Principal/PrincipalRole/CatalogRole+grants,
> > the
> > > > > parts
> > > > > > > that
> > > > > > > > > end
> > > > > > > > > > > up
> > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > the underlying authz just reuse existing
> > Authorizer
> > > > > logic.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Based on some other offline discussion with
> > > Dmitri, I
> > > > > > > > realized
> > > > > > > > > > why
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > discussion around "requirements" may have been
> > > > > confusing:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. My attempts to shorten the doc inadvertently
> > cut
> > > > out
> > > > > > > some
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > "abstract requirements" formulations
> > > > > > > > > > > > > > > 2. The project in itself is somewhat more of a
> > > "form
> > > > > *as*
> > > > > > > > > > function"
> > > > > > > > > > > > > > > feature than *strictly* functional in nature --
> > for
> > > > > > > example,
> > > > > > > > > many
> > > > > > > > > > > > > > correctly
> > > > > > > > > > > > > > > pointed out that "sharing" via the existing
> > > > > RBAC/Principals
> > > > > > > > > > syntax
> > > > > > > > > > > is
> > > > > > > > > > > > > > > already *possible* just by letting a cross-org
> > > > > "consumer"
> > > > > > > be
> > > > > > > > a
> > > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > principal and access your own company's main
> > > Polaris
> > > > > > > catalog
> > > > > > > > > > > > endpoints.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For (1) I added this to the Requirements
> section
> > in
> > > > > the doc
> > > > > > > > to
> > > > > > > > > > > better
> > > > > > > > > > > > > > > clarify the abstract form of the data model
> > > element:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *A. Sharing Container*: Container concept for
> > > holding
> > > > > > > things
> > > > > > > > > you
> > > > > > > > > > > > want to
> > > > > > > > > > > > > > > share - “ShareEntity”
> > > > > > > > > > > > > > > *B. Consumer Identity*: Special flavor of
> > > > > principal-like
> > > > > > > > entity
> > > > > > > > > > > > > > > representing a consumer relationship -
> > > > > “ExternalConsumer”
> > > > > > > > > > > > > > > *C. Map Containers to Consumers*: A way to
> > specify
> > > > > letting
> > > > > > > a
> > > > > > > > > > given
> > > > > > > > > > > > > > > consumer access a given sharing container -
> > > “Listing”
> > > > > > > > > > > > > > > *D. Consumer Endpoint*: A segregated API
> > > access-point
> > > > > for
> > > > > > > > > > > consumers -
> > > > > > > > > > > > > > > “EndpointConfig”
> > > > > > > > > > > > > > > *E. Objects in Containers*: A way to specify
> > which
> > > > > > > in-catalog
> > > > > > > > > > > objects
> > > > > > > > > > > > > > > belong in the big sharing container -
> > > > “ShareMembership”
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This still bakes in a bit of implementation
> > detail
> > > > > since it
> > > > > > > > > begs
> > > > > > > > > > > the
> > > > > > > > > > > > > > > question why not just let core RBAC/grants
> handle
> > > (C)
> > > > > and
> > > > > > > > (E).
> > > > > > > > > To
> > > > > > > > > > > > that
> > > > > > > > > > > > > > end,
> > > > > > > > > > > > > > > and addressing problem (2) of where the "form
> > *is*
> > > > > > > > function", I
> > > > > > > > > > > > added a
> > > > > > > > > > > > > > > "Concept Mapping" tab:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think the details of more advanced features
> can
> > > > > still be
> > > > > > > > > hashed
> > > > > > > > > > > out
> > > > > > > > > > > > > > > incrementally without blocking work on an MVP
> > (and
> > > > > > > certainly
> > > > > > > > > the
> > > > > > > > > > > MVP
> > > > > > > > > > > > > > scope
> > > > > > > > > > > > > > > could be trimmed so we can launch-and-iterate),
> > so
> > > as
> > > > > long
> > > > > > > as
> > > > > > > > > > > there's
> > > > > > > > > > > > > > > community alignment on the general direction I
> > > > propose
> > > > > we
> > > > > > > > could
> > > > > > > > > > > move
> > > > > > > > > > > > > > > forward on basic building blocks. Or if
> building
> > > > blocks
> > > > > > > > > > themselves
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > controversial we could still proceed with some
> > > > > prototyping
> > > > > > > > > across
> > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > options.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste
> > > > Onofré <
> > > > > > > > > > > > jb@nanthrax.net>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Hi everyone,
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> I would like to resume the discussion about
> Data
> > > > > Sharing
> > > > > > > in
> > > > > > > > > > > Polaris
> > > > > > > > > > > > > > >> and propose focusing on three major items to
> > help
> > > us
> > > > > move
> > > > > > > > > > forward.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> 1. What is data sharing?
> > > > > > > > > > > > > > >> In a nutshell, it is an open protocol
> > proposal
> > > > > > > allowing
> > > > > > > > > > > > > > >> organizations to share live data without
> copying
> > > or
> > > > > > > > > replication.
> > > > > > > > > > > The
> > > > > > > > > > > > > > >> main goal is zero-copy sharing achieved
> through
> > > > shared
> > > > > > > > > metadata
> > > > > > > > > > > and
> > > > > > > > > > > > > > >> authorization.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Polaris, this would mean:
> > > > > > > > > > > > > > >> a. The consumer is authenticated as usual.
> > > > > > > > > > > > > > >> b. The Polaris server provides the consumer
> with
> > > > > direct
> > > > > > > > access
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >> data files (e.g., Parquet, as specified in the
> > > > > > > > metadata.json).
> > > > > > > > > > > > > > >> c. The consumer reads the data files directly
> > > using
> > > > > their
> > > > > > > > > > > preferred
> > > > > > > > > > > > > > >> tool (e.g., a query engine).
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> While Polaris already has most of the
> necessary
> > > > > underlying
> > > > > > > > > > > > > > >> capabilities, we need clear branding/packaging
> > > > around
> > > > > this
> > > > > > > > use
> > > > > > > > > > > case.
> > > > > > > > > > > > > > >> Specifically, this means:
> > > > > > > > > > > > > > >> a. A share is a dedicated catalog in the
> Polaris
> > > > > server,
> > > > > > > or
> > > > > > > > a
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > >> role scoped to specific namespaces.
> > > > > > > > > > > > > > >> b. A recipient is represented by a principal
> > and a
> > > > > > > principal
> > > > > > > > > > role.
> > > > > > > > > > > > > > >> c. A consumer profile consists of the catalog
> > URI
> > > > and
> > > > > the
> > > > > > > > > OAuth2
> > > > > > > > > > > > > > >> client_id/client_secret.
> > > > > > > > > > > > > > >> d. We use credential vending (temporary STS
> > > > > credentials)
> > > > > > > for
> > > > > > > > > > > > consumer
> > > > > > > > > > > > > > >> access to the data files.
> > > > > > > > > > > > > > >> e. We use the Polaris IRC layer to expose the
> > > > > metadata and
> > > > > > > > the
> > > > > > > > > > > file
> > > > > > > > > > > > > > >> locations/access.
> > > > > > > > > > > > > > >> f. We can also support Polaris-to-Polaris
> > sharing
> > > > via
> > > > > > > > > > Federation.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Concretely:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> - On the provider side, we use the standard
> > > > Polaris
> > > > > RBAC
> > > > > > > > > model
> > > > > > > > > > > > > > >> (assigning grants to catalog roles, and
> > assigning
> > > > > those
> > > > > > > > > catalog
> > > > > > > > > > > > roles
> > > > > > > > > > > > > > >> to principal roles):
> > > > > > > > > > > > > > >> - Create a dedicated principal with its
> > own
> > > > > > > > > > > > > > client_id/client_secret.
> > > > > > > > > > > > > > >> - Create a catalog role with
> > > TABLE_READ_DATA,
> > > > > > > > > TABLE_LIST,
> > > > > > > > > > > and
> > > > > > > > > > > > > > >> NAMESPACE_LIST grants on the namespaces to be
> > > shared
> > > > > (no
> > > > > > > > write
> > > > > > > > > > > > > > >> grants).
> > > > > > > > > > > > > > >> - Create a principal role per
> destination,
> > > > > attach
> > > > > > > the
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > >> role, and assign the principal.
> > > > > > > > > > > > > > >> - The destination engine uses loadTable
> > with
> > > > the
> > > > > > > > > > > > > > >> "X-Iceberg-Access-Delegation:
> > vended-credentials"
> > > > > header.
> > > > > > > > > > Polaris
> > > > > > > > > > > > then
> > > > > > > > > > > > > > >> emits temporary, read-only storage
> credentials,
> > > > > allowing
> > > > > > > the
> > > > > > > > > > > engine
> > > > > > > > > > > > to
> > > > > > > > > > > > > > >> read the data files directly from the owner's
> > > bucket
> > > > > > > without
> > > > > > > > > > > > accessing
> > > > > > > > > > > > > > >> the rest of the bucket. This achieves
> zero-copy
> > > > > sharing.
> > > > > > > > > > > > > > >> - On the recipient/consumer side, no special
> > > > > > > configuration
> > > > > > > > > is
> > > > > > > > > > > > > > >> needed; any IRC client can use it.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> 2. What is data sharing not?
> > > > > > > > > > > > > > >> I previously considered adding an Arrow
> > Flight
> > > > > > > endpoint
> > > > > > > > to
> > > > > > > > > > > > > > >> Polaris, but I no longer think it is a good
> idea
> > > for
> > > > > the
> > > > > > > > > initial
> > > > > > > > > > > > > > >> phase. An Arrow Flight endpoint would serve
> the
> > > data
> > > > > > > > directly,
> > > > > > > > > > > which
> > > > > > > > > > > > > > >> would require Polaris to read the data and
> > expose
> > > > it.
> > > > > I am
> > > > > > > > > > against
> > > > > > > > > > > > > > >> embedding a query engine in Polaris, as this
> > tight
> > > > > > > coupling
> > > > > > > > > > feels
> > > > > > > > > > > > like
> > > > > > > > > > > > > > >> an anti-pattern. While we could use a
> delegation
> > > > > service
> > > > > > > to
> > > > > > > > > > serve
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >> Flight endpoint, I think it should be
> considered
> > > > for a
> > > > > > > later
> > > > > > > > > > > phase,
> > > > > > > > > > > > if
> > > > > > > > > > > > > > >> at all, since a Flight endpoint might be
> better
> > > > > suited as
> > > > > > > a
> > > > > > > > > > > consumer
> > > > > > > > > > > > > > >> responsibility and could encourage data
> copying.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> The first phase should focus on packaging and
> > > > > branding the
> > > > > > > > > > > > > > >> capabilities we already have.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> 3. What could the Data Sharing "package" look
> > > like
> > > > in
> > > > > > > > > Polaris?
> > > > > > > > > > > > > > >> To make Data Sharing obvious and easy to
> > use,
> > > we
> > > > > > > should
> > > > > > > > > > > > introduce
> > > > > > > > > > > > > > >> the missing concept of a "share"—a named
> > > definition
> > > > > that
> > > > > > > > users
> > > > > > > > > > can
> > > > > > > > > > > > > > >> list, discover, and define. The goal is to
> > > > > materialize the
> > > > > > > > > > user's
> > > > > > > > > > > > > > >> intent ("I want to share these tables with
> this
> > > > > consumer
> > > > > > > > until
> > > > > > > > > > > this
> > > > > > > > > > > > > > >> date") into standard Polaris entities
> > (principal,
> > > > > > > principal
> > > > > > > > > > role,
> > > > > > > > > > > > > > >> catalog role, and grants).
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Therefore, I propose adding a /shares endpoint
> > to
> > > > the
> > > > > > > > Polaris
> > > > > > > > > > > > > > >> Management API to package and materialize
> these
> > > > share
> > > > > > > > > > definitions
> > > > > > > > > > > > into
> > > > > > > > > > > > > > >> Polaris entities. This would provide a usable,
> > > > > > > first-version
> > > > > > > > > > > package
> > > > > > > > > > > > > > >> for Polaris Data Sharing.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> I hope this addresses Dmitri's questions and
> > > > provides
> > > > > a
> > > > > > > more
> > > > > > > > > > > > concrete
> > > > > > > > > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Thoughts?
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Regards,
> > > > > > > > > > > > > > >> JB
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <
> > > > > > > dhuo@apache.org
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > Hi All,
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > One big emerging enterprise use case coming
> up
> > > as
> > > > > more
> > > > > > > > > people
> > > > > > > > > > > > > > >> consolidate
> > > > > > > > > > > > > > >> > Data Lakehouses and Catalogs is something
> > > commonly
> > > > > known
> > > > > > > > as
> > > > > > > > > > > "Data
> > > > > > > > > > > > > > >> Sharing",
> > > > > > > > > > > > > > >> > an more specifically over the course of
> > adoption
> > > > of
> > > > > open
> > > > > > > > > table
> > > > > > > > > > > > formats
> > > > > > > > > > > > > > >> > "Open Data Sharing".
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > Examples of existing managed service
> > providers'
> > > > Data
> > > > > > > > Sharing
> > > > > > > > > > > > features:
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > https://www.databricks.com/product/delta-sharing
> > > > > > > > > > > > > > >> >
> > > > > > > > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > The basic idea is that when you share data
> > > between
> > > > > > > > different
> > > > > > > > > > > > > > companies,
> > > > > > > > > > > > > > >> you
> > > > > > > > > > > > > > >> > need a first-class governance/management
> layer
> > > and
> > > > > extra
> > > > > > > > > > > > > > >> bells-and-whistles
> > > > > > > > > > > > > > >> > that are distinct from just the basic
> > > capabilities
> > > > > of
> > > > > > > RBAC
> > > > > > > > > or
> > > > > > > > > > > > > > >> generalized
> > > > > > > > > > > > > > >> > access-control (i.e. if you're sharing
> across
> > > > > > > > > > > partially-untrusted
> > > > > > > > > > > > org
> > > > > > > > > > > > > > >> > boundaries, you don't just let the consumer
> > > > > organization
> > > > > > > > log
> > > > > > > > > > > into
> > > > > > > > > > > > your
> > > > > > > > > > > > > > >> > datalake like one of your own employees).
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > JB and I put together this high-level
> proposal
> > > for
> > > > > > > > > supporting
> > > > > > > > > > > Open
> > > > > > > > > > > > > > >> Sharing
> > > > > > > > > > > > > > >> > in Polaris:
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > Tentatively, it means adding ~5 logical data
> > > model
> > > > > > > > > constructs,
> > > > > > > > > > > > some of
> > > > > > > > > > > > > > >> > which may be a first-class PolarisEntity
> type,
> > > > > others
> > > > > > > > > subtypes
> > > > > > > > > > > of
> > > > > > > > > > > > > > >> existing
> > > > > > > > > > > > > > >> > entities, and others just a nested
> construct:
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > - ShareEntity (would behave similarly to
> a
> > > > > Catalog)
> > > > > > > > > > > > > > >> > - ExternalConsumer (mostly inherits from
> > > > > Principal)
> > > > > > > > > > > > > > >> > - Listing (Similar to a "role grant" but
> > has
> > > > > > > different
> > > > > > > > > > > > metadata)
> > > > > > > > > > > > > > >> > - EndpointConfig (nested config under
> > > Listing)
> > > > > > > > > > > > > > >> > - ShareMembership (Similar to a
> "securable
> > > > > grant" but
> > > > > > > > > > > different
> > > > > > > > > > > > > > >> metadata)
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > Feedback/comments welcome! I'll also bring
> it
> > up
> > > > for
> > > > > > > live
> > > > > > > > > > > > discussion
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > >> > there's time in the community sync.
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > Cheers,
> > > > > > > > > > > > > > >> > Dennis
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dennis Huo <dh...@apache.org>.
Ah thanks for clarifying - to be clear, the ConsumerCredential is *not*
strictly necessary for the Listing in terms of spec expressibility, only
the ExternalConsumer is.
An ExternalConsumer doesn't need to be backed by a ConsumerCredential; the
ExternalConsumer is an abstract Sharing-specific pointer to whatever does
the indirection in the sharing implementation.
Responded to your github comment - I'll split anything that touches
"ConsumerCredential" into a separate yaml file so it's easier to see the
core auth-agnostic sharing spec in one file, with the more
implementation-specific decorations around ConsumerCredential in an
auxiliary file.
I suppose another option is to even split it into separate PRs (in addition
to separate yaml files?). Then we can easily see one commit that's just the
pure abstract auth-agnostic API, that in theory private Polaris
implementations could already take and implement on external-IdP
architectures to support the same agnostic spec.
On Wed, Oct 21, 2026 at 1:54 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi Dennis,
>
> While I still support the idea of splitting OpenAPI yaml files and URI
> bases between the Management and Sharing APIs, I do not think it is tightly
> coupled with my proposal for the authentication-neural Sharing API spec.
>
> To give an example, currently [5446] CreateListingRequest includes a
> "consumerName". This immediately binds the Listing definition to
> an ExternalConsumer and by extension to ConsumerCredential.
>
> ConsumerCredential is naturally part of the authentication process during
> Share access. I believe ConsumerCredential (and related entities) is worth
> isolating from the Shares API to increase interoperability.
>
> Implementations that choose to rely on ConsumerCredential for Share access
> can still do so. However, ConsumerCredential will not be a requirement for
> all possible Share API implementations.
>
> This can work with any OpenAPI yaml and URI paths layout.
>
> I'd like to resolve this aspect before we go back to URI paths and YAML
> file locations. Actually, I do not feel too strongly about path prefixes. I
> do believe separate prefixes make implementations easier in many aspects
> (including network setup), but interleaving those paths with the Management
> API is not a blocker, from my POV, only an inconvenience.
>
> Cheers,
> Dmitri.
>
> [5446] https://github.com/apache/polaris/pull/5446
>
> On Wed, Oct 21, 2026 at 3:19 PM Dennis Huo <dh...@apache.org> wrote:
>
> > Yeah, that's a good topic to discuss; I guess it touches on our core
> > (unresolved?) conventions on how we organize API url-prefixes and yaml
> > files. Reposting my response to the github comment as well:
> >
> > Right, I think we could certainly split the API routes and
> yaml-definitions
> > even further if we want to ensure a well-defined auth-detail-free core,
> but
> > for the standalone-Polaris use case we'd still need to define it
> > *somewhere*
> > under a Sharing-specific surface, right?
> >
> > @dimas-b <https://github.com/dimas-b> Were you thinking of just
> splitting
> > the yaml file but keeping the share-management root, or adding a third
> > sharing prefix like
> >
> >
> /api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials...
> > ?
> >
> > Or just yaml-file split would be keeping
> > /api/shares-management/v1/external-consumers/{name}/credentials... but
> > putting it in a separate yaml file to make it easy for service-runners to
> > choose not to support that whole yaml file's concrete APIs.
> >
> > On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi All,
> > >
> > > Re: the shating API PR [5446], I'm repeating my GH comment here for
> > > visibility.
> > >
> > > I'd like the API to be neutral from the authentication perspective.
> > >
> > > This is how we handle the IRC spec in Polaris. The API declares that
> > > clients need to authenticate, but specific details about principals,
> > > clients and secrets are delegated to the server.
> > >
> > > I think this is a sound approach for the sharing API too. Specific
> > details
> > > related to native Polaris principals (ConsumerCredential, etc.) can
> > > probably remain in the Management API.
> > >
> > > WDYT?
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > [5446] https://github.com/apache/polaris/pull/5446
> > >
> > > On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > As I said during the community meeting, let's focus on the API PR
> > > > first. We will build from there.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <dh...@apache.org> wrote:
> > > > >
> > > > > +1, thanks for the discussion everyone!
> > > > >
> > > > > Yes, I meant to flip it to "ready to review" earlier; flipped it
> now.
> > > I'm
> > > > > still working through the existing set of comments and may need to
> > come
> > > > > back to it in the next few days before getting proper replies, but
> > feel
> > > > > free to add additional comments.
> > > > >
> > > > > Recapping the community sync discussion re: Dmitri's last message,
> > I'm
> > > > > fully aligned on building the sharing on top of existing authz
> > > > mechanisms,
> > > > > instead of being a new mechanism, and indeed external IdP is
> intended
> > > to
> > > > be
> > > > > a first-class supported use case.
> > > > >
> > > > > The latest distinction is that the ExternalConsumer isn't
> > fundamentally
> > > > > identified by any internal PolarisPrincipal, but rather that it
> > serves
> > > > as a
> > > > > wrapper that references some identity, whether that identity is
> > defined
> > > > in
> > > > > an external IdP without any latent in-Polaris representation or if
> > > it's a
> > > > > hidden PolarisPrincipal (for the Polaris-standalone RBAC use case).
> > > > >
> > > > > The sharing endpoint via ExternalConsumer is the only layer of
> > > > indirection
> > > > > at the API entry-point, but then once the set of
> activePrincipalRoles
> > > is
> > > > > resolved, the underlying authz engine doesn't behave any
> differently
> > > from
> > > > > normal Polaris authz.
> > > > >
> > > > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <fl...@gmail.com>
> > wrote:
> > > > >
> > > > > > Thanks everyone for the discussion!
> > > > > >
> > > > > > As a next step discussed in today's community sync, we will start
> > > > reviewing
> > > > > > the spec change.
> > > > > >
> > > > > > Hi Dennis, is this PR ready for review?
> > > > > > https://github.com/apache/polaris/pull/5446
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > >
> > > > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <
> > > travismbowen@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for this discussion Dennis!
> > > > > > >
> > > > > > > Wanted to weigh in with my thoughts and questions.
> > > > > > >
> > > > > > > >
> > > > > > > > Do you prefer /api/shares-management/v1/
> > > > > > > > instead of adding to /api/management/v1?
> > > > > > >
> > > > > > >
> > > > > > > While I can see the argument for a share specific management
> > > endpoint
> > > > > > there
> > > > > > > doesn't seem to be a clear obvious choice from what I can see
> - I
> > > > lean
> > > > > > > towards /api/management as being the central place for all the
> > > > > > management -
> > > > > > > share or not. As long as we're careful not to leak share types
> or
> > > non
> > > > > > share
> > > > > > > types into the wrong api (which could be helped by having
> > separate
> > > > yamls)
> > > > > > > then it feels reasonable to have all under the management api.
> > > > > > >
> > > > > > > Also went through the doc and PR and made several comments as
> > well.
> > > > > > >
> > > > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > Prithvi mentioned ExternalConsumer and principals. I'd like
> to
> > > > > > > > reinforce the need to further discuss "consumers" and
> > principals
> > > > and
> > > > > > how
> > > > > > > > they relate to IdP and Polaris Authentication.
> > > > > > > >
> > > > > > > > From my POV, share access control should not be a new
> mechanism
> > > but
> > > > > > > should
> > > > > > > > build on top of the existing extensible authentication /
> > > > authorization
> > > > > > > > framework. Supporting external IdP for shares would be a
> > valuable
> > > > > > feature
> > > > > > > > IMHO.
> > > > > > > >
> > > > > > > > Side note: I'm posting comments on PR 5446 too.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <
> > > > prithvisivasankar@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > Thank you for reviving this, and for the concrete spec in
> PR
> > > > 5446..
> > > > > > the
> > > > > > > > > journeys doc was especially helpful for seeing the
> end-to-end
> > > > flow.
> > > > > > > > >
> > > > > > > > > I agree this is worth doing, and that v1 is more than a
> > > tutorial
> > > > on
> > > > > > > > today's
> > > > > > > > > RBAC. A partner principal on /api/catalog is already
> > possible;
> > > > what
> > > > > > > seems
> > > > > > > > > valuable is a first-class share (enumerated members, a
> > > restricted
> > > > > > > > consumer,
> > > > > > > > > a listing as the binding and audit unit) plus a segregated,
> > > > read-only
> > > > > > > > > Iceberg REST surface with credential vending. Stock Iceberg
> > > REST
> > > > as
> > > > > > the
> > > > > > > > > consumer protocol, and keeping Flight and
> Polaris-to-Polaris
> > > > > > federation
> > > > > > > > out
> > > > > > > > > of v1, all sound right to me.
> > > > > > > > >
> > > > > > > > > I don't have a strong view yet on the URL layout
> (management
> > vs
> > > > > > shares
> > > > > > > > > prefix, nested vs top-level, prefix keyed by share vs
> > listing)
> > > > and am
> > > > > > > > happy
> > > > > > > > > to follow whatever the community prefers there.
> > > > > > > > >
> > > > > > > > > A few things I wanted to check on the spec itself, several
> of
> > > > these
> > > > > > > echo
> > > > > > > > > points already raised on the list:
> > > > > > > > >
> > > > > > > > > 1. Membership as source of truth. If a hidden catalog role
> is
> > > > > > > > independently
> > > > > > > > > grantable, a share could be widened without going through
> the
> > > > share
> > > > > > > APIs.
> > > > > > > > > Would it make sense to say the data plane authorizes
> against
> > > > > > > membership +
> > > > > > > > > listing, and that any backing roles are not user-visible /
> > not
> > > > > > > > > independently grantable?
> > > > > > > > > 2. ExternalConsumer and principals. The Aug 7 recap already
> > > > noted we
> > > > > > > > should
> > > > > > > > > not assume a PrincipalEntity behind every ExternalConsumer.
> > The
> > > > > > > journeys
> > > > > > > > > still describe granting the consumer a share principal
> role,
> > I
> > > > assume
> > > > > > > > that
> > > > > > > > > is an implementation sketch for v1 client_id/secret, not
> the
> > > > public
> > > > > > > > model?
> > > > > > > > > 3. Listing expiration. JB's original note was "share these
> > > tables
> > > > > > with
> > > > > > > > this
> > > > > > > > > consumer until this date." Credential expiry is a bit
> > > different.
> > > > > > Would
> > > > > > > a
> > > > > > > > > listing-level expiration (and maybe ACTIVE/SUSPENDED) be in
> > > > scope for
> > > > > > > v1?
> > > > > > > > > 4. Time-travel. As raised in the community sync, sharing
> via
> > > > Iceberg
> > > > > > > REST
> > > > > > > > > means historical snapshots remain readable. Totally fine
> for
> > v1
> > > > to
> > > > > > > > support
> > > > > > > > > only full history, would an explicit field on the share
> help
> > > so a
> > > > > > later
> > > > > > > > > CURRENT_ONLY option is not a breaking surprise?
> > > > > > > > > 5. Consumer data plane. The current file is control-plane
> > only,
> > > > which
> > > > > > > is
> > > > > > > > a
> > > > > > > > > good first step. It might help to list the Iceberg routes
> > that
> > > > are
> > > > > > > > mounted,
> > > > > > > > > that writes 404, that non-members 404, and that /config
> does
> > > not
> > > > > > return
> > > > > > > > > provider catalog properties, so implementations don't
> > diverge.
> > > > > > > > >
> > > > > > > > > Two smaller notes, if useful: in the J1 journey the
> consumer
> > > > still
> > > > > > > posts
> > > > > > > > to
> > > > > > > > > /api/catalog/v1/oauth/tokens, if partners are not supposed
> to
> > > > touch
> > > > > > the
> > > > > > > > > internal catalog API, should the default token endpoint
> live
> > on
> > > > the
> > > > > > > > shares
> > > > > > > > > surface (or an external IdP)? And on Dmitri's point about
> > share
> > > > > > > operators
> > > > > > > > > vs catalog operators, a SHARE_MANAGE privilege on the bound
> > > > catalog
> > > > > > > > > (independent of CATALOG_MANAGE_ACCESS) would make that work
> > on
> > > > any
> > > > > > URL
> > > > > > > > > prefix.
> > > > > > > > >
> > > > > > > > > Happy to review the next revision, and thanks again to both
> > of
> > > > you
> > > > > > for
> > > > > > > > > driving this.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Prithvi S
> > > > > > > > >
> > > > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <dhuo@apache.org
> >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > As discussed live, looks like we both might have some
> > drafts
> > > > we can
> > > > > > > > > compare
> > > > > > > > > > and choose/merge from :)
> > > > > > > > > >
> > > > > > > > > > Here's my current draft spec PR; it opts for my preferred
> > > > syntax
> > > > > > > > choices
> > > > > > > > > > for now (/api/management/ for share-management,
> > /api/shares/
> > > > for
> > > > > > > > consumer
> > > > > > > > > > data-plane): https://github.com/apache/polaris/pull/5446
> > > > > > > > > >
> > > > > > > > > > Since we were talking about preferring mdfiles for design
> > > > iteration
> > > > > > > > > here's
> > > > > > > > > > a companion design-addendum which just enumerates what
> the
> > > > > > end-to-end
> > > > > > > > > user
> > > > > > > > > > journeys look like syntactically for each combination of
> > > design
> > > > > > > > choices:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > > > > > > > >
> > > > > > > > > > Please let me know if anyone prefers a Google Doc version
> > of
> > > > the
> > > > > > > > > > design-addendum for easier commenting, or if the
> > > > design-addendum
> > > > > > > itself
> > > > > > > > > > should also be modeled as a PR on some free-floating
> branch
> > > > just to
> > > > > > > > > capture
> > > > > > > > > > comments.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <
> > > > > > jb@nanthrax.net
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > I am resuming work on this proposal.
> > > > > > > > > > >
> > > > > > > > > > > Regarding our recent discussion, I will move forward
> with
> > > > > > opening a
> > > > > > > > > > > draft PR for the /api/shares/v1 endpoint to help
> > illustrate
> > > > the
> > > > > > > > > > > design.
> > > > > > > > > > >
> > > > > > > > > > > I will update this thread once the PR is ready.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > JB
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <
> > > > > > > dimas@apache.org
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Dennis,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the recap!
> > > > > > > > > > > >
> > > > > > > > > > > > I was thinking that share management is probably
> > distinct
> > > > from
> > > > > > > > > catalog
> > > > > > > > > > > > management. A user who already has a realm+catalog
> may
> > > > have an
> > > > > > > > option
> > > > > > > > > > to
> > > > > > > > > > > > manage shares, but not the catalog itself. Would it
> > make
> > > > sense
> > > > > > to
> > > > > > > > > > > separate
> > > > > > > > > > > > the share management API from the general Polaris
> > > > management
> > > > > > > API? I
> > > > > > > > > > mean
> > > > > > > > > > > > using a different prefix for URIs.
> > > > > > > > > > > >
> > > > > > > > > > > > For example: /api/shares/v1/
> > > > > > > > > > > >
> > > > > > > > > > > > Obviously clients should not make any assumptions
> about
> > > the
> > > > > > base
> > > > > > > > path
> > > > > > > > > > > > (/api/shares), which could be different in different
> > > > > > deployments
> > > > > > > > > (e.g.
> > > > > > > > > > > > going through a gateway). Clients must be able to
> take
> > > the
> > > > base
> > > > > > > URI
> > > > > > > > > as
> > > > > > > > > > a
> > > > > > > > > > > > config option. Path segments after /v1/ will be
> > specified
> > > > by
> > > > > > > > Polaris
> > > > > > > > > > > (Open
> > > > > > > > > > > > API yaml).
> > > > > > > > > > > >
> > > > > > > > > > > > WDYT?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Dmitri.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <
> > > dhuo@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Just to recap some of the discussion from today's
> > > > community
> > > > > > > sync,
> > > > > > > > > > > Dmitri,
> > > > > > > > > > > > > Robert, and Alex helped raise a few considerations
> to
> > > > roll
> > > > > > into
> > > > > > > > the
> > > > > > > > > > > next
> > > > > > > > > > > > > design/prototyping phase:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Given the way sharing is packaged as a separate
> > > > thing, it
> > > > > > > may
> > > > > > > > > not
> > > > > > > > > > be
> > > > > > > > > > > > > obvious to users that historical snapshots are
> > equally
> > > > > > > available
> > > > > > > > to
> > > > > > > > > > > > > consumers as the latest snapshot (by virtue of
> > sharing
> > > > via
> > > > > > the
> > > > > > > > > > > standard IRC
> > > > > > > > > > > > > protocol) - we should make sure to document that
> fact
> > > > well,
> > > > > > > > > including
> > > > > > > > > > > the
> > > > > > > > > > > > > pitfall that deleting rows by updating a table
> > doesn't
> > > > > > > > necessarily
> > > > > > > > > > mean
> > > > > > > > > > > > > those rows become inaccessible to consumers
> > > > > > > > > > > > > 2. Make sure not to hard-code assumptions about the
> > > > existence
> > > > > > > of
> > > > > > > > an
> > > > > > > > > > > > > internal PrincipalEntity backing an
> ExternalConsumer
> > -
> > > > even
> > > > > > if
> > > > > > > > the
> > > > > > > > > > > first
> > > > > > > > > > > > > MVP implementation uses a nested/hidden Principal
> of
> > > some
> > > > > > type
> > > > > > > > that
> > > > > > > > > > > holds a
> > > > > > > > > > > > > Polaris-managed client_id/client_secret, the
> > 3rd-party
> > > > IdP
> > > > > > use
> > > > > > > > > cases
> > > > > > > > > > > > > require "implicit" Principals resolved via the
> oauth
> > > > token.
> > > > > > In
> > > > > > > > such
> > > > > > > > > > > cases
> > > > > > > > > > > > > the ExternalConsumer contains the metadata
> > coordinating
> > > > the
> > > > > > > > > three-way
> > > > > > > > > > > > > handshake between the data-provider, data-consumer,
> > and
> > > > IdP,
> > > > > > > but
> > > > > > > > > > there
> > > > > > > > > > > is
> > > > > > > > > > > > > no PrincipalEntity underlying it.
> > > > > > > > > > > > > 3. Longer-term, we may want to tackle
> "on-behalf-of"
> > > > > > semantics
> > > > > > > -
> > > > > > > > > > > initially,
> > > > > > > > > > > > > the way an ExternalConsumer works is that the
> > > > finer-grained
> > > > > > > > > consumer
> > > > > > > > > > > users
> > > > > > > > > > > > > share a single ExternalConsumer identity, just like
> > > how a
> > > > > > > > FEDERATED
> > > > > > > > > > > catalog
> > > > > > > > > > > > > shares a single service principal in its
> > > > ConnectionConfig.
> > > > > > But
> > > > > > > in
> > > > > > > > > > some
> > > > > > > > > > > > > cases the consumer may want some end-to-end
> > attribution
> > > > of
> > > > > > > > > > fine-grained
> > > > > > > > > > > > > users, like "on-behalf-of: bob @
> consumercompany.com
> > "
> > > > when
> > > > > > the
> > > > > > > > > > > > > ExternalConsumer implicit principal is something
> like
> > > > > > > > > > > > > "externalconsumer$consumercompany"
> > > > > > > > > > > > > 4. We can iterate more rapidly on a draft spec PR
> and
> > > > also
> > > > > > > expand
> > > > > > > > > on
> > > > > > > > > > > the
> > > > > > > > > > > > > detailed technical design
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Dennis
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <
> > > > dhuo@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks JB for reviving this thread, and belated
> > > thanks
> > > > to
> > > > > > all
> > > > > > > > > > > reviewers
> > > > > > > > > > > > > > who have left feedback in the doc or this thread
> so
> > > > far!
> > > > > > And
> > > > > > > > > > > apologies
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > the silence on this, it's still been on my radar
> > but
> > > I
> > > > got
> > > > > > > > > > > distracted by
> > > > > > > > > > > > > > some other work for awhile.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Echoing JB's summary, the proposal is indeed that
> > > we're
> > > > > > > mostly
> > > > > > > > > just
> > > > > > > > > > > > > > reusing existing Authz/RBAC machinery, and the
> > > > translation
> > > > > > > > layer
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > sharing concepts sits *on top of* the existing
> > authz
> > > > engine
> > > > > > > as
> > > > > > > > > much
> > > > > > > > > > > as
> > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So even if we're adding layers of indirection on
> > > > > > > > > > > ExternalConsumer/Listing
> > > > > > > > > > > > > > -> Principal/PrincipalRole/CatalogRole+grants,
> the
> > > > parts
> > > > > > that
> > > > > > > > end
> > > > > > > > > > up
> > > > > > > > > > > > > doing
> > > > > > > > > > > > > > the underlying authz just reuse existing
> Authorizer
> > > > logic.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Based on some other offline discussion with
> > Dmitri, I
> > > > > > > realized
> > > > > > > > > why
> > > > > > > > > > > the
> > > > > > > > > > > > > > discussion around "requirements" may have been
> > > > confusing:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. My attempts to shorten the doc inadvertently
> cut
> > > out
> > > > > > some
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > more
> > > > > > > > > > > > > > "abstract requirements" formulations
> > > > > > > > > > > > > > 2. The project in itself is somewhat more of a
> > "form
> > > > *as*
> > > > > > > > > function"
> > > > > > > > > > > > > > feature than *strictly* functional in nature --
> for
> > > > > > example,
> > > > > > > > many
> > > > > > > > > > > > > correctly
> > > > > > > > > > > > > > pointed out that "sharing" via the existing
> > > > RBAC/Principals
> > > > > > > > > syntax
> > > > > > > > > > is
> > > > > > > > > > > > > > already *possible* just by letting a cross-org
> > > > "consumer"
> > > > > > be
> > > > > > > a
> > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > principal and access your own company's main
> > Polaris
> > > > > > catalog
> > > > > > > > > > > endpoints.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For (1) I added this to the Requirements section
> in
> > > > the doc
> > > > > > > to
> > > > > > > > > > better
> > > > > > > > > > > > > > clarify the abstract form of the data model
> > element:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *A. Sharing Container*: Container concept for
> > holding
> > > > > > things
> > > > > > > > you
> > > > > > > > > > > want to
> > > > > > > > > > > > > > share - “ShareEntity”
> > > > > > > > > > > > > > *B. Consumer Identity*: Special flavor of
> > > > principal-like
> > > > > > > entity
> > > > > > > > > > > > > > representing a consumer relationship -
> > > > “ExternalConsumer”
> > > > > > > > > > > > > > *C. Map Containers to Consumers*: A way to
> specify
> > > > letting
> > > > > > a
> > > > > > > > > given
> > > > > > > > > > > > > > consumer access a given sharing container -
> > “Listing”
> > > > > > > > > > > > > > *D. Consumer Endpoint*: A segregated API
> > access-point
> > > > for
> > > > > > > > > > consumers -
> > > > > > > > > > > > > > “EndpointConfig”
> > > > > > > > > > > > > > *E. Objects in Containers*: A way to specify
> which
> > > > > > in-catalog
> > > > > > > > > > objects
> > > > > > > > > > > > > > belong in the big sharing container -
> > > “ShareMembership”
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This still bakes in a bit of implementation
> detail
> > > > since it
> > > > > > > > begs
> > > > > > > > > > the
> > > > > > > > > > > > > > question why not just let core RBAC/grants handle
> > (C)
> > > > and
> > > > > > > (E).
> > > > > > > > To
> > > > > > > > > > > that
> > > > > > > > > > > > > end,
> > > > > > > > > > > > > > and addressing problem (2) of where the "form
> *is*
> > > > > > > function", I
> > > > > > > > > > > added a
> > > > > > > > > > > > > > "Concept Mapping" tab:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think the details of more advanced features can
> > > > still be
> > > > > > > > hashed
> > > > > > > > > > out
> > > > > > > > > > > > > > incrementally without blocking work on an MVP
> (and
> > > > > > certainly
> > > > > > > > the
> > > > > > > > > > MVP
> > > > > > > > > > > > > scope
> > > > > > > > > > > > > > could be trimmed so we can launch-and-iterate),
> so
> > as
> > > > long
> > > > > > as
> > > > > > > > > > there's
> > > > > > > > > > > > > > community alignment on the general direction I
> > > propose
> > > > we
> > > > > > > could
> > > > > > > > > > move
> > > > > > > > > > > > > > forward on basic building blocks. Or if building
> > > blocks
> > > > > > > > > themselves
> > > > > > > > > > > are
> > > > > > > > > > > > > > controversial we could still proceed with some
> > > > prototyping
> > > > > > > > across
> > > > > > > > > > > > > different
> > > > > > > > > > > > > > options.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > Dennis
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste
> > > Onofré <
> > > > > > > > > > > jb@nanthrax.net>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Hi everyone,
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> I would like to resume the discussion about Data
> > > > Sharing
> > > > > > in
> > > > > > > > > > Polaris
> > > > > > > > > > > > > >> and propose focusing on three major items to
> help
> > us
> > > > move
> > > > > > > > > forward.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> 1. What is data sharing?
> > > > > > > > > > > > > >> In a nutshell, it is an open protocol
> proposal
> > > > > > allowing
> > > > > > > > > > > > > >> organizations to share live data without copying
> > or
> > > > > > > > replication.
> > > > > > > > > > The
> > > > > > > > > > > > > >> main goal is zero-copy sharing achieved through
> > > shared
> > > > > > > > metadata
> > > > > > > > > > and
> > > > > > > > > > > > > >> authorization.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Polaris, this would mean:
> > > > > > > > > > > > > >> a. The consumer is authenticated as usual.
> > > > > > > > > > > > > >> b. The Polaris server provides the consumer with
> > > > direct
> > > > > > > access
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > >> data files (e.g., Parquet, as specified in the
> > > > > > > metadata.json).
> > > > > > > > > > > > > >> c. The consumer reads the data files directly
> > using
> > > > their
> > > > > > > > > > preferred
> > > > > > > > > > > > > >> tool (e.g., a query engine).
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> While Polaris already has most of the necessary
> > > > underlying
> > > > > > > > > > > > > >> capabilities, we need clear branding/packaging
> > > around
> > > > this
> > > > > > > use
> > > > > > > > > > case.
> > > > > > > > > > > > > >> Specifically, this means:
> > > > > > > > > > > > > >> a. A share is a dedicated catalog in the Polaris
> > > > server,
> > > > > > or
> > > > > > > a
> > > > > > > > > > > catalog
> > > > > > > > > > > > > >> role scoped to specific namespaces.
> > > > > > > > > > > > > >> b. A recipient is represented by a principal
> and a
> > > > > > principal
> > > > > > > > > role.
> > > > > > > > > > > > > >> c. A consumer profile consists of the catalog
> URI
> > > and
> > > > the
> > > > > > > > OAuth2
> > > > > > > > > > > > > >> client_id/client_secret.
> > > > > > > > > > > > > >> d. We use credential vending (temporary STS
> > > > credentials)
> > > > > > for
> > > > > > > > > > > consumer
> > > > > > > > > > > > > >> access to the data files.
> > > > > > > > > > > > > >> e. We use the Polaris IRC layer to expose the
> > > > metadata and
> > > > > > > the
> > > > > > > > > > file
> > > > > > > > > > > > > >> locations/access.
> > > > > > > > > > > > > >> f. We can also support Polaris-to-Polaris
> sharing
> > > via
> > > > > > > > > Federation.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Concretely:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> - On the provider side, we use the standard
> > > Polaris
> > > > RBAC
> > > > > > > > model
> > > > > > > > > > > > > >> (assigning grants to catalog roles, and
> assigning
> > > > those
> > > > > > > > catalog
> > > > > > > > > > > roles
> > > > > > > > > > > > > >> to principal roles):
> > > > > > > > > > > > > >> - Create a dedicated principal with its
> own
> > > > > > > > > > > > > client_id/client_secret.
> > > > > > > > > > > > > >> - Create a catalog role with
> > TABLE_READ_DATA,
> > > > > > > > TABLE_LIST,
> > > > > > > > > > and
> > > > > > > > > > > > > >> NAMESPACE_LIST grants on the namespaces to be
> > shared
> > > > (no
> > > > > > > write
> > > > > > > > > > > > > >> grants).
> > > > > > > > > > > > > >> - Create a principal role per destination,
> > > > attach
> > > > > > the
> > > > > > > > > > catalog
> > > > > > > > > > > > > >> role, and assign the principal.
> > > > > > > > > > > > > >> - The destination engine uses loadTable
> with
> > > the
> > > > > > > > > > > > > >> "X-Iceberg-Access-Delegation:
> vended-credentials"
> > > > header.
> > > > > > > > > Polaris
> > > > > > > > > > > then
> > > > > > > > > > > > > >> emits temporary, read-only storage credentials,
> > > > allowing
> > > > > > the
> > > > > > > > > > engine
> > > > > > > > > > > to
> > > > > > > > > > > > > >> read the data files directly from the owner's
> > bucket
> > > > > > without
> > > > > > > > > > > accessing
> > > > > > > > > > > > > >> the rest of the bucket. This achieves zero-copy
> > > > sharing.
> > > > > > > > > > > > > >> - On the recipient/consumer side, no special
> > > > > > configuration
> > > > > > > > is
> > > > > > > > > > > > > >> needed; any IRC client can use it.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> 2. What is data sharing not?
> > > > > > > > > > > > > >> I previously considered adding an Arrow
> Flight
> > > > > > endpoint
> > > > > > > to
> > > > > > > > > > > > > >> Polaris, but I no longer think it is a good idea
> > for
> > > > the
> > > > > > > > initial
> > > > > > > > > > > > > >> phase. An Arrow Flight endpoint would serve the
> > data
> > > > > > > directly,
> > > > > > > > > > which
> > > > > > > > > > > > > >> would require Polaris to read the data and
> expose
> > > it.
> > > > I am
> > > > > > > > > against
> > > > > > > > > > > > > >> embedding a query engine in Polaris, as this
> tight
> > > > > > coupling
> > > > > > > > > feels
> > > > > > > > > > > like
> > > > > > > > > > > > > >> an anti-pattern. While we could use a delegation
> > > > service
> > > > > > to
> > > > > > > > > serve
> > > > > > > > > > > the
> > > > > > > > > > > > > >> Flight endpoint, I think it should be considered
> > > for a
> > > > > > later
> > > > > > > > > > phase,
> > > > > > > > > > > if
> > > > > > > > > > > > > >> at all, since a Flight endpoint might be better
> > > > suited as
> > > > > > a
> > > > > > > > > > consumer
> > > > > > > > > > > > > >> responsibility and could encourage data copying.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> The first phase should focus on packaging and
> > > > branding the
> > > > > > > > > > > > > >> capabilities we already have.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> 3. What could the Data Sharing "package" look
> > like
> > > in
> > > > > > > > Polaris?
> > > > > > > > > > > > > >> To make Data Sharing obvious and easy to
> use,
> > we
> > > > > > should
> > > > > > > > > > > introduce
> > > > > > > > > > > > > >> the missing concept of a "share"—a named
> > definition
> > > > that
> > > > > > > users
> > > > > > > > > can
> > > > > > > > > > > > > >> list, discover, and define. The goal is to
> > > > materialize the
> > > > > > > > > user's
> > > > > > > > > > > > > >> intent ("I want to share these tables with this
> > > > consumer
> > > > > > > until
> > > > > > > > > > this
> > > > > > > > > > > > > >> date") into standard Polaris entities
> (principal,
> > > > > > principal
> > > > > > > > > role,
> > > > > > > > > > > > > >> catalog role, and grants).
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Therefore, I propose adding a /shares endpoint
> to
> > > the
> > > > > > > Polaris
> > > > > > > > > > > > > >> Management API to package and materialize these
> > > share
> > > > > > > > > definitions
> > > > > > > > > > > into
> > > > > > > > > > > > > >> Polaris entities. This would provide a usable,
> > > > > > first-version
> > > > > > > > > > package
> > > > > > > > > > > > > >> for Polaris Data Sharing.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> I hope this addresses Dmitri's questions and
> > > provides
> > > > a
> > > > > > more
> > > > > > > > > > > concrete
> > > > > > > > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Thoughts?
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Regards,
> > > > > > > > > > > > > >> JB
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <
> > > > > > dhuo@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Hi All,
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > One big emerging enterprise use case coming up
> > as
> > > > more
> > > > > > > > people
> > > > > > > > > > > > > >> consolidate
> > > > > > > > > > > > > >> > Data Lakehouses and Catalogs is something
> > commonly
> > > > known
> > > > > > > as
> > > > > > > > > > "Data
> > > > > > > > > > > > > >> Sharing",
> > > > > > > > > > > > > >> > an more specifically over the course of
> adoption
> > > of
> > > > open
> > > > > > > > table
> > > > > > > > > > > formats
> > > > > > > > > > > > > >> > "Open Data Sharing".
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Examples of existing managed service
> providers'
> > > Data
> > > > > > > Sharing
> > > > > > > > > > > features:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > https://www.databricks.com/product/delta-sharing
> > > > > > > > > > > > > >> >
> > > > > > > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > The basic idea is that when you share data
> > between
> > > > > > > different
> > > > > > > > > > > > > companies,
> > > > > > > > > > > > > >> you
> > > > > > > > > > > > > >> > need a first-class governance/management layer
> > and
> > > > extra
> > > > > > > > > > > > > >> bells-and-whistles
> > > > > > > > > > > > > >> > that are distinct from just the basic
> > capabilities
> > > > of
> > > > > > RBAC
> > > > > > > > or
> > > > > > > > > > > > > >> generalized
> > > > > > > > > > > > > >> > access-control (i.e. if you're sharing across
> > > > > > > > > > partially-untrusted
> > > > > > > > > > > org
> > > > > > > > > > > > > >> > boundaries, you don't just let the consumer
> > > > organization
> > > > > > > log
> > > > > > > > > > into
> > > > > > > > > > > your
> > > > > > > > > > > > > >> > datalake like one of your own employees).
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > JB and I put together this high-level proposal
> > for
> > > > > > > > supporting
> > > > > > > > > > Open
> > > > > > > > > > > > > >> Sharing
> > > > > > > > > > > > > >> > in Polaris:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Tentatively, it means adding ~5 logical data
> > model
> > > > > > > > constructs,
> > > > > > > > > > > some of
> > > > > > > > > > > > > >> > which may be a first-class PolarisEntity type,
> > > > others
> > > > > > > > subtypes
> > > > > > > > > > of
> > > > > > > > > > > > > >> existing
> > > > > > > > > > > > > >> > entities, and others just a nested construct:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > - ShareEntity (would behave similarly to a
> > > > Catalog)
> > > > > > > > > > > > > >> > - ExternalConsumer (mostly inherits from
> > > > Principal)
> > > > > > > > > > > > > >> > - Listing (Similar to a "role grant" but
> has
> > > > > > different
> > > > > > > > > > > metadata)
> > > > > > > > > > > > > >> > - EndpointConfig (nested config under
> > Listing)
> > > > > > > > > > > > > >> > - ShareMembership (Similar to a "securable
> > > > grant" but
> > > > > > > > > > different
> > > > > > > > > > > > > >> metadata)
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Feedback/comments welcome! I'll also bring it
> up
> > > for
> > > > > > live
> > > > > > > > > > > discussion
> > > > > > > > > > > > > if
> > > > > > > > > > > > > >> > there's time in the community sync.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Cheers,
> > > > > > > > > > > > > >> > Dennis
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Dennis,
While I still support the idea of splitting OpenAPI yaml files and URI
bases between the Management and Sharing APIs, I do not think it is tightly
coupled with my proposal for the authentication-neural Sharing API spec.
To give an example, currently [5446] CreateListingRequest includes a
"consumerName". This immediately binds the Listing definition to
an ExternalConsumer and by extension to ConsumerCredential.
ConsumerCredential is naturally part of the authentication process during
Share access. I believe ConsumerCredential (and related entities) is worth
isolating from the Shares API to increase interoperability.
Implementations that choose to rely on ConsumerCredential for Share access
can still do so. However, ConsumerCredential will not be a requirement for
all possible Share API implementations.
This can work with any OpenAPI yaml and URI paths layout.
I'd like to resolve this aspect before we go back to URI paths and YAML
file locations. Actually, I do not feel too strongly about path prefixes. I
do believe separate prefixes make implementations easier in many aspects
(including network setup), but interleaving those paths with the Management
API is not a blocker, from my POV, only an inconvenience.
Cheers,
Dmitri.
[5446] https://github.com/apache/polaris/pull/5446
On Wed, Oct 21, 2026 at 3:19 PM Dennis Huo <dh...@apache.org> wrote:
> Yeah, that's a good topic to discuss; I guess it touches on our core
> (unresolved?) conventions on how we organize API url-prefixes and yaml
> files. Reposting my response to the github comment as well:
>
> Right, I think we could certainly split the API routes and yaml-definitions
> even further if we want to ensure a well-defined auth-detail-free core, but
> for the standalone-Polaris use case we'd still need to define it
> *somewhere*
> under a Sharing-specific surface, right?
>
> @dimas-b <https://github.com/dimas-b> Were you thinking of just splitting
> the yaml file but keeping the share-management root, or adding a third
> sharing prefix like
>
> /api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials...
> ?
>
> Or just yaml-file split would be keeping
> /api/shares-management/v1/external-consumers/{name}/credentials... but
> putting it in a separate yaml file to make it easy for service-runners to
> choose not to support that whole yaml file's concrete APIs.
>
> On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > Re: the shating API PR [5446], I'm repeating my GH comment here for
> > visibility.
> >
> > I'd like the API to be neutral from the authentication perspective.
> >
> > This is how we handle the IRC spec in Polaris. The API declares that
> > clients need to authenticate, but specific details about principals,
> > clients and secrets are delegated to the server.
> >
> > I think this is a sound approach for the sharing API too. Specific
> details
> > related to native Polaris principals (ConsumerCredential, etc.) can
> > probably remain in the Management API.
> >
> > WDYT?
> >
> > Thanks,
> > Dmitri.
> >
> > [5446] https://github.com/apache/polaris/pull/5446
> >
> > On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi
> > >
> > > As I said during the community meeting, let's focus on the API PR
> > > first. We will build from there.
> > >
> > > Regards
> > > JB
> > >
> > > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <dh...@apache.org> wrote:
> > > >
> > > > +1, thanks for the discussion everyone!
> > > >
> > > > Yes, I meant to flip it to "ready to review" earlier; flipped it now.
> > I'm
> > > > still working through the existing set of comments and may need to
> come
> > > > back to it in the next few days before getting proper replies, but
> feel
> > > > free to add additional comments.
> > > >
> > > > Recapping the community sync discussion re: Dmitri's last message,
> I'm
> > > > fully aligned on building the sharing on top of existing authz
> > > mechanisms,
> > > > instead of being a new mechanism, and indeed external IdP is intended
> > to
> > > be
> > > > a first-class supported use case.
> > > >
> > > > The latest distinction is that the ExternalConsumer isn't
> fundamentally
> > > > identified by any internal PolarisPrincipal, but rather that it
> serves
> > > as a
> > > > wrapper that references some identity, whether that identity is
> defined
> > > in
> > > > an external IdP without any latent in-Polaris representation or if
> > it's a
> > > > hidden PolarisPrincipal (for the Polaris-standalone RBAC use case).
> > > >
> > > > The sharing endpoint via ExternalConsumer is the only layer of
> > > indirection
> > > > at the API entry-point, but then once the set of activePrincipalRoles
> > is
> > > > resolved, the underlying authz engine doesn't behave any differently
> > from
> > > > normal Polaris authz.
> > > >
> > > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > >
> > > > > Thanks everyone for the discussion!
> > > > >
> > > > > As a next step discussed in today's community sync, we will start
> > > reviewing
> > > > > the spec change.
> > > > >
> > > > > Hi Dennis, is this PR ready for review?
> > > > > https://github.com/apache/polaris/pull/5446
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <
> > travismbowen@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks for this discussion Dennis!
> > > > > >
> > > > > > Wanted to weigh in with my thoughts and questions.
> > > > > >
> > > > > > >
> > > > > > > Do you prefer /api/shares-management/v1/
> > > > > > > instead of adding to /api/management/v1?
> > > > > >
> > > > > >
> > > > > > While I can see the argument for a share specific management
> > endpoint
> > > > > there
> > > > > > doesn't seem to be a clear obvious choice from what I can see - I
> > > lean
> > > > > > towards /api/management as being the central place for all the
> > > > > management -
> > > > > > share or not. As long as we're careful not to leak share types or
> > non
> > > > > share
> > > > > > types into the wrong api (which could be helped by having
> separate
> > > yamls)
> > > > > > then it feels reasonable to have all under the management api.
> > > > > >
> > > > > > Also went through the doc and PR and made several comments as
> well.
> > > > > >
> > > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Prithvi mentioned ExternalConsumer and principals. I'd like to
> > > > > > > reinforce the need to further discuss "consumers" and
> principals
> > > and
> > > > > how
> > > > > > > they relate to IdP and Polaris Authentication.
> > > > > > >
> > > > > > > From my POV, share access control should not be a new mechanism
> > but
> > > > > > should
> > > > > > > build on top of the existing extensible authentication /
> > > authorization
> > > > > > > framework. Supporting external IdP for shares would be a
> valuable
> > > > > feature
> > > > > > > IMHO.
> > > > > > >
> > > > > > > Side note: I'm posting comments on PR 5446 too.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <
> > > prithvisivasankar@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > Thank you for reviving this, and for the concrete spec in PR
> > > 5446..
> > > > > the
> > > > > > > > journeys doc was especially helpful for seeing the end-to-end
> > > flow.
> > > > > > > >
> > > > > > > > I agree this is worth doing, and that v1 is more than a
> > tutorial
> > > on
> > > > > > > today's
> > > > > > > > RBAC. A partner principal on /api/catalog is already
> possible;
> > > what
> > > > > > seems
> > > > > > > > valuable is a first-class share (enumerated members, a
> > restricted
> > > > > > > consumer,
> > > > > > > > a listing as the binding and audit unit) plus a segregated,
> > > read-only
> > > > > > > > Iceberg REST surface with credential vending. Stock Iceberg
> > REST
> > > as
> > > > > the
> > > > > > > > consumer protocol, and keeping Flight and Polaris-to-Polaris
> > > > > federation
> > > > > > > out
> > > > > > > > of v1, all sound right to me.
> > > > > > > >
> > > > > > > > I don't have a strong view yet on the URL layout (management
> vs
> > > > > shares
> > > > > > > > prefix, nested vs top-level, prefix keyed by share vs
> listing)
> > > and am
> > > > > > > happy
> > > > > > > > to follow whatever the community prefers there.
> > > > > > > >
> > > > > > > > A few things I wanted to check on the spec itself, several of
> > > these
> > > > > > echo
> > > > > > > > points already raised on the list:
> > > > > > > >
> > > > > > > > 1. Membership as source of truth. If a hidden catalog role is
> > > > > > > independently
> > > > > > > > grantable, a share could be widened without going through the
> > > share
> > > > > > APIs.
> > > > > > > > Would it make sense to say the data plane authorizes against
> > > > > > membership +
> > > > > > > > listing, and that any backing roles are not user-visible /
> not
> > > > > > > > independently grantable?
> > > > > > > > 2. ExternalConsumer and principals. The Aug 7 recap already
> > > noted we
> > > > > > > should
> > > > > > > > not assume a PrincipalEntity behind every ExternalConsumer.
> The
> > > > > > journeys
> > > > > > > > still describe granting the consumer a share principal role,
> I
> > > assume
> > > > > > > that
> > > > > > > > is an implementation sketch for v1 client_id/secret, not the
> > > public
> > > > > > > model?
> > > > > > > > 3. Listing expiration. JB's original note was "share these
> > tables
> > > > > with
> > > > > > > this
> > > > > > > > consumer until this date." Credential expiry is a bit
> > different.
> > > > > Would
> > > > > > a
> > > > > > > > listing-level expiration (and maybe ACTIVE/SUSPENDED) be in
> > > scope for
> > > > > > v1?
> > > > > > > > 4. Time-travel. As raised in the community sync, sharing via
> > > Iceberg
> > > > > > REST
> > > > > > > > means historical snapshots remain readable. Totally fine for
> v1
> > > to
> > > > > > > support
> > > > > > > > only full history, would an explicit field on the share help
> > so a
> > > > > later
> > > > > > > > CURRENT_ONLY option is not a breaking surprise?
> > > > > > > > 5. Consumer data plane. The current file is control-plane
> only,
> > > which
> > > > > > is
> > > > > > > a
> > > > > > > > good first step. It might help to list the Iceberg routes
> that
> > > are
> > > > > > > mounted,
> > > > > > > > that writes 404, that non-members 404, and that /config does
> > not
> > > > > return
> > > > > > > > provider catalog properties, so implementations don't
> diverge.
> > > > > > > >
> > > > > > > > Two smaller notes, if useful: in the J1 journey the consumer
> > > still
> > > > > > posts
> > > > > > > to
> > > > > > > > /api/catalog/v1/oauth/tokens, if partners are not supposed to
> > > touch
> > > > > the
> > > > > > > > internal catalog API, should the default token endpoint live
> on
> > > the
> > > > > > > shares
> > > > > > > > surface (or an external IdP)? And on Dmitri's point about
> share
> > > > > > operators
> > > > > > > > vs catalog operators, a SHARE_MANAGE privilege on the bound
> > > catalog
> > > > > > > > (independent of CATALOG_MANAGE_ACCESS) would make that work
> on
> > > any
> > > > > URL
> > > > > > > > prefix.
> > > > > > > >
> > > > > > > > Happy to review the next revision, and thanks again to both
> of
> > > you
> > > > > for
> > > > > > > > driving this.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Prithvi S
> > > > > > > >
> > > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <dh...@apache.org>
> > > wrote:
> > > > > > > >
> > > > > > > > > As discussed live, looks like we both might have some
> drafts
> > > we can
> > > > > > > > compare
> > > > > > > > > and choose/merge from :)
> > > > > > > > >
> > > > > > > > > Here's my current draft spec PR; it opts for my preferred
> > > syntax
> > > > > > > choices
> > > > > > > > > for now (/api/management/ for share-management,
> /api/shares/
> > > for
> > > > > > > consumer
> > > > > > > > > data-plane): https://github.com/apache/polaris/pull/5446
> > > > > > > > >
> > > > > > > > > Since we were talking about preferring mdfiles for design
> > > iteration
> > > > > > > > here's
> > > > > > > > > a companion design-addendum which just enumerates what the
> > > > > end-to-end
> > > > > > > > user
> > > > > > > > > journeys look like syntactically for each combination of
> > design
> > > > > > > choices:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > > > > > > >
> > > > > > > > > Please let me know if anyone prefers a Google Doc version
> of
> > > the
> > > > > > > > > design-addendum for easier commenting, or if the
> > > design-addendum
> > > > > > itself
> > > > > > > > > should also be modeled as a PR on some free-floating branch
> > > just to
> > > > > > > > capture
> > > > > > > > > comments.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <
> > > > > jb@nanthrax.net
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi everyone,
> > > > > > > > > >
> > > > > > > > > > I am resuming work on this proposal.
> > > > > > > > > >
> > > > > > > > > > Regarding our recent discussion, I will move forward with
> > > > > opening a
> > > > > > > > > > draft PR for the /api/shares/v1 endpoint to help
> illustrate
> > > the
> > > > > > > > > > design.
> > > > > > > > > >
> > > > > > > > > > I will update this thread once the PR is ready.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > JB
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Dennis,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the recap!
> > > > > > > > > > >
> > > > > > > > > > > I was thinking that share management is probably
> distinct
> > > from
> > > > > > > > catalog
> > > > > > > > > > > management. A user who already has a realm+catalog may
> > > have an
> > > > > > > option
> > > > > > > > > to
> > > > > > > > > > > manage shares, but not the catalog itself. Would it
> make
> > > sense
> > > > > to
> > > > > > > > > > separate
> > > > > > > > > > > the share management API from the general Polaris
> > > management
> > > > > > API? I
> > > > > > > > > mean
> > > > > > > > > > > using a different prefix for URIs.
> > > > > > > > > > >
> > > > > > > > > > > For example: /api/shares/v1/
> > > > > > > > > > >
> > > > > > > > > > > Obviously clients should not make any assumptions about
> > the
> > > > > base
> > > > > > > path
> > > > > > > > > > > (/api/shares), which could be different in different
> > > > > deployments
> > > > > > > > (e.g.
> > > > > > > > > > > going through a gateway). Clients must be able to take
> > the
> > > base
> > > > > > URI
> > > > > > > > as
> > > > > > > > > a
> > > > > > > > > > > config option. Path segments after /v1/ will be
> specified
> > > by
> > > > > > > Polaris
> > > > > > > > > > (Open
> > > > > > > > > > > API yaml).
> > > > > > > > > > >
> > > > > > > > > > > WDYT?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <
> > dhuo@apache.org
> > > >
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Just to recap some of the discussion from today's
> > > community
> > > > > > sync,
> > > > > > > > > > Dmitri,
> > > > > > > > > > > > Robert, and Alex helped raise a few considerations to
> > > roll
> > > > > into
> > > > > > > the
> > > > > > > > > > next
> > > > > > > > > > > > design/prototyping phase:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Given the way sharing is packaged as a separate
> > > thing, it
> > > > > > may
> > > > > > > > not
> > > > > > > > > be
> > > > > > > > > > > > obvious to users that historical snapshots are
> equally
> > > > > > available
> > > > > > > to
> > > > > > > > > > > > consumers as the latest snapshot (by virtue of
> sharing
> > > via
> > > > > the
> > > > > > > > > > standard IRC
> > > > > > > > > > > > protocol) - we should make sure to document that fact
> > > well,
> > > > > > > > including
> > > > > > > > > > the
> > > > > > > > > > > > pitfall that deleting rows by updating a table
> doesn't
> > > > > > > necessarily
> > > > > > > > > mean
> > > > > > > > > > > > those rows become inaccessible to consumers
> > > > > > > > > > > > 2. Make sure not to hard-code assumptions about the
> > > existence
> > > > > > of
> > > > > > > an
> > > > > > > > > > > > internal PrincipalEntity backing an ExternalConsumer
> -
> > > even
> > > > > if
> > > > > > > the
> > > > > > > > > > first
> > > > > > > > > > > > MVP implementation uses a nested/hidden Principal of
> > some
> > > > > type
> > > > > > > that
> > > > > > > > > > holds a
> > > > > > > > > > > > Polaris-managed client_id/client_secret, the
> 3rd-party
> > > IdP
> > > > > use
> > > > > > > > cases
> > > > > > > > > > > > require "implicit" Principals resolved via the oauth
> > > token.
> > > > > In
> > > > > > > such
> > > > > > > > > > cases
> > > > > > > > > > > > the ExternalConsumer contains the metadata
> coordinating
> > > the
> > > > > > > > three-way
> > > > > > > > > > > > handshake between the data-provider, data-consumer,
> and
> > > IdP,
> > > > > > but
> > > > > > > > > there
> > > > > > > > > > is
> > > > > > > > > > > > no PrincipalEntity underlying it.
> > > > > > > > > > > > 3. Longer-term, we may want to tackle "on-behalf-of"
> > > > > semantics
> > > > > > -
> > > > > > > > > > initially,
> > > > > > > > > > > > the way an ExternalConsumer works is that the
> > > finer-grained
> > > > > > > > consumer
> > > > > > > > > > users
> > > > > > > > > > > > share a single ExternalConsumer identity, just like
> > how a
> > > > > > > FEDERATED
> > > > > > > > > > catalog
> > > > > > > > > > > > shares a single service principal in its
> > > ConnectionConfig.
> > > > > But
> > > > > > in
> > > > > > > > > some
> > > > > > > > > > > > cases the consumer may want some end-to-end
> attribution
> > > of
> > > > > > > > > fine-grained
> > > > > > > > > > > > users, like "on-behalf-of: bob @ consumercompany.com
> "
> > > when
> > > > > the
> > > > > > > > > > > > ExternalConsumer implicit principal is something like
> > > > > > > > > > > > "externalconsumer$consumercompany"
> > > > > > > > > > > > 4. We can iterate more rapidly on a draft spec PR and
> > > also
> > > > > > expand
> > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > > > detailed technical design
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Dennis
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <
> > > dhuo@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks JB for reviving this thread, and belated
> > thanks
> > > to
> > > > > all
> > > > > > > > > > reviewers
> > > > > > > > > > > > > who have left feedback in the doc or this thread so
> > > far!
> > > > > And
> > > > > > > > > > apologies
> > > > > > > > > > > > for
> > > > > > > > > > > > > the silence on this, it's still been on my radar
> but
> > I
> > > got
> > > > > > > > > > distracted by
> > > > > > > > > > > > > some other work for awhile.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Echoing JB's summary, the proposal is indeed that
> > we're
> > > > > > mostly
> > > > > > > > just
> > > > > > > > > > > > > reusing existing Authz/RBAC machinery, and the
> > > translation
> > > > > > > layer
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > new
> > > > > > > > > > > > > sharing concepts sits *on top of* the existing
> authz
> > > engine
> > > > > > as
> > > > > > > > much
> > > > > > > > > > as
> > > > > > > > > > > > > possible.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So even if we're adding layers of indirection on
> > > > > > > > > > ExternalConsumer/Listing
> > > > > > > > > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the
> > > parts
> > > > > that
> > > > > > > end
> > > > > > > > > up
> > > > > > > > > > > > doing
> > > > > > > > > > > > > the underlying authz just reuse existing Authorizer
> > > logic.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Based on some other offline discussion with
> Dmitri, I
> > > > > > realized
> > > > > > > > why
> > > > > > > > > > the
> > > > > > > > > > > > > discussion around "requirements" may have been
> > > confusing:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. My attempts to shorten the doc inadvertently cut
> > out
> > > > > some
> > > > > > of
> > > > > > > > the
> > > > > > > > > > more
> > > > > > > > > > > > > "abstract requirements" formulations
> > > > > > > > > > > > > 2. The project in itself is somewhat more of a
> "form
> > > *as*
> > > > > > > > function"
> > > > > > > > > > > > > feature than *strictly* functional in nature -- for
> > > > > example,
> > > > > > > many
> > > > > > > > > > > > correctly
> > > > > > > > > > > > > pointed out that "sharing" via the existing
> > > RBAC/Principals
> > > > > > > > syntax
> > > > > > > > > is
> > > > > > > > > > > > > already *possible* just by letting a cross-org
> > > "consumer"
> > > > > be
> > > > > > a
> > > > > > > > > > > > first-class
> > > > > > > > > > > > > principal and access your own company's main
> Polaris
> > > > > catalog
> > > > > > > > > > endpoints.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For (1) I added this to the Requirements section in
> > > the doc
> > > > > > to
> > > > > > > > > better
> > > > > > > > > > > > > clarify the abstract form of the data model
> element:
> > > > > > > > > > > > >
> > > > > > > > > > > > > *A. Sharing Container*: Container concept for
> holding
> > > > > things
> > > > > > > you
> > > > > > > > > > want to
> > > > > > > > > > > > > share - “ShareEntity”
> > > > > > > > > > > > > *B. Consumer Identity*: Special flavor of
> > > principal-like
> > > > > > entity
> > > > > > > > > > > > > representing a consumer relationship -
> > > “ExternalConsumer”
> > > > > > > > > > > > > *C. Map Containers to Consumers*: A way to specify
> > > letting
> > > > > a
> > > > > > > > given
> > > > > > > > > > > > > consumer access a given sharing container -
> “Listing”
> > > > > > > > > > > > > *D. Consumer Endpoint*: A segregated API
> access-point
> > > for
> > > > > > > > > consumers -
> > > > > > > > > > > > > “EndpointConfig”
> > > > > > > > > > > > > *E. Objects in Containers*: A way to specify which
> > > > > in-catalog
> > > > > > > > > objects
> > > > > > > > > > > > > belong in the big sharing container -
> > “ShareMembership”
> > > > > > > > > > > > >
> > > > > > > > > > > > > This still bakes in a bit of implementation detail
> > > since it
> > > > > > > begs
> > > > > > > > > the
> > > > > > > > > > > > > question why not just let core RBAC/grants handle
> (C)
> > > and
> > > > > > (E).
> > > > > > > To
> > > > > > > > > > that
> > > > > > > > > > > > end,
> > > > > > > > > > > > > and addressing problem (2) of where the "form *is*
> > > > > > function", I
> > > > > > > > > > added a
> > > > > > > > > > > > > "Concept Mapping" tab:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think the details of more advanced features can
> > > still be
> > > > > > > hashed
> > > > > > > > > out
> > > > > > > > > > > > > incrementally without blocking work on an MVP (and
> > > > > certainly
> > > > > > > the
> > > > > > > > > MVP
> > > > > > > > > > > > scope
> > > > > > > > > > > > > could be trimmed so we can launch-and-iterate), so
> as
> > > long
> > > > > as
> > > > > > > > > there's
> > > > > > > > > > > > > community alignment on the general direction I
> > propose
> > > we
> > > > > > could
> > > > > > > > > move
> > > > > > > > > > > > > forward on basic building blocks. Or if building
> > blocks
> > > > > > > > themselves
> > > > > > > > > > are
> > > > > > > > > > > > > controversial we could still proceed with some
> > > prototyping
> > > > > > > across
> > > > > > > > > > > > different
> > > > > > > > > > > > > options.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Dennis
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste
> > Onofré <
> > > > > > > > > > jb@nanthrax.net>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Hi everyone,
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> I would like to resume the discussion about Data
> > > Sharing
> > > > > in
> > > > > > > > > Polaris
> > > > > > > > > > > > >> and propose focusing on three major items to help
> us
> > > move
> > > > > > > > forward.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> 1. What is data sharing?
> > > > > > > > > > > > >> In a nutshell, it is an open protocol proposal
> > > > > allowing
> > > > > > > > > > > > >> organizations to share live data without copying
> or
> > > > > > > replication.
> > > > > > > > > The
> > > > > > > > > > > > >> main goal is zero-copy sharing achieved through
> > shared
> > > > > > > metadata
> > > > > > > > > and
> > > > > > > > > > > > >> authorization.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Polaris, this would mean:
> > > > > > > > > > > > >> a. The consumer is authenticated as usual.
> > > > > > > > > > > > >> b. The Polaris server provides the consumer with
> > > direct
> > > > > > access
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > >> data files (e.g., Parquet, as specified in the
> > > > > > metadata.json).
> > > > > > > > > > > > >> c. The consumer reads the data files directly
> using
> > > their
> > > > > > > > > preferred
> > > > > > > > > > > > >> tool (e.g., a query engine).
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> While Polaris already has most of the necessary
> > > underlying
> > > > > > > > > > > > >> capabilities, we need clear branding/packaging
> > around
> > > this
> > > > > > use
> > > > > > > > > case.
> > > > > > > > > > > > >> Specifically, this means:
> > > > > > > > > > > > >> a. A share is a dedicated catalog in the Polaris
> > > server,
> > > > > or
> > > > > > a
> > > > > > > > > > catalog
> > > > > > > > > > > > >> role scoped to specific namespaces.
> > > > > > > > > > > > >> b. A recipient is represented by a principal and a
> > > > > principal
> > > > > > > > role.
> > > > > > > > > > > > >> c. A consumer profile consists of the catalog URI
> > and
> > > the
> > > > > > > OAuth2
> > > > > > > > > > > > >> client_id/client_secret.
> > > > > > > > > > > > >> d. We use credential vending (temporary STS
> > > credentials)
> > > > > for
> > > > > > > > > > consumer
> > > > > > > > > > > > >> access to the data files.
> > > > > > > > > > > > >> e. We use the Polaris IRC layer to expose the
> > > metadata and
> > > > > > the
> > > > > > > > > file
> > > > > > > > > > > > >> locations/access.
> > > > > > > > > > > > >> f. We can also support Polaris-to-Polaris sharing
> > via
> > > > > > > > Federation.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Concretely:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> - On the provider side, we use the standard
> > Polaris
> > > RBAC
> > > > > > > model
> > > > > > > > > > > > >> (assigning grants to catalog roles, and assigning
> > > those
> > > > > > > catalog
> > > > > > > > > > roles
> > > > > > > > > > > > >> to principal roles):
> > > > > > > > > > > > >> - Create a dedicated principal with its own
> > > > > > > > > > > > client_id/client_secret.
> > > > > > > > > > > > >> - Create a catalog role with
> TABLE_READ_DATA,
> > > > > > > TABLE_LIST,
> > > > > > > > > and
> > > > > > > > > > > > >> NAMESPACE_LIST grants on the namespaces to be
> shared
> > > (no
> > > > > > write
> > > > > > > > > > > > >> grants).
> > > > > > > > > > > > >> - Create a principal role per destination,
> > > attach
> > > > > the
> > > > > > > > > catalog
> > > > > > > > > > > > >> role, and assign the principal.
> > > > > > > > > > > > >> - The destination engine uses loadTable with
> > the
> > > > > > > > > > > > >> "X-Iceberg-Access-Delegation: vended-credentials"
> > > header.
> > > > > > > > Polaris
> > > > > > > > > > then
> > > > > > > > > > > > >> emits temporary, read-only storage credentials,
> > > allowing
> > > > > the
> > > > > > > > > engine
> > > > > > > > > > to
> > > > > > > > > > > > >> read the data files directly from the owner's
> bucket
> > > > > without
> > > > > > > > > > accessing
> > > > > > > > > > > > >> the rest of the bucket. This achieves zero-copy
> > > sharing.
> > > > > > > > > > > > >> - On the recipient/consumer side, no special
> > > > > configuration
> > > > > > > is
> > > > > > > > > > > > >> needed; any IRC client can use it.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> 2. What is data sharing not?
> > > > > > > > > > > > >> I previously considered adding an Arrow Flight
> > > > > endpoint
> > > > > > to
> > > > > > > > > > > > >> Polaris, but I no longer think it is a good idea
> for
> > > the
> > > > > > > initial
> > > > > > > > > > > > >> phase. An Arrow Flight endpoint would serve the
> data
> > > > > > directly,
> > > > > > > > > which
> > > > > > > > > > > > >> would require Polaris to read the data and expose
> > it.
> > > I am
> > > > > > > > against
> > > > > > > > > > > > >> embedding a query engine in Polaris, as this tight
> > > > > coupling
> > > > > > > > feels
> > > > > > > > > > like
> > > > > > > > > > > > >> an anti-pattern. While we could use a delegation
> > > service
> > > > > to
> > > > > > > > serve
> > > > > > > > > > the
> > > > > > > > > > > > >> Flight endpoint, I think it should be considered
> > for a
> > > > > later
> > > > > > > > > phase,
> > > > > > > > > > if
> > > > > > > > > > > > >> at all, since a Flight endpoint might be better
> > > suited as
> > > > > a
> > > > > > > > > consumer
> > > > > > > > > > > > >> responsibility and could encourage data copying.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> The first phase should focus on packaging and
> > > branding the
> > > > > > > > > > > > >> capabilities we already have.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> 3. What could the Data Sharing "package" look
> like
> > in
> > > > > > > Polaris?
> > > > > > > > > > > > >> To make Data Sharing obvious and easy to use,
> we
> > > > > should
> > > > > > > > > > introduce
> > > > > > > > > > > > >> the missing concept of a "share"—a named
> definition
> > > that
> > > > > > users
> > > > > > > > can
> > > > > > > > > > > > >> list, discover, and define. The goal is to
> > > materialize the
> > > > > > > > user's
> > > > > > > > > > > > >> intent ("I want to share these tables with this
> > > consumer
> > > > > > until
> > > > > > > > > this
> > > > > > > > > > > > >> date") into standard Polaris entities (principal,
> > > > > principal
> > > > > > > > role,
> > > > > > > > > > > > >> catalog role, and grants).
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Therefore, I propose adding a /shares endpoint to
> > the
> > > > > > Polaris
> > > > > > > > > > > > >> Management API to package and materialize these
> > share
> > > > > > > > definitions
> > > > > > > > > > into
> > > > > > > > > > > > >> Polaris entities. This would provide a usable,
> > > > > first-version
> > > > > > > > > package
> > > > > > > > > > > > >> for Polaris Data Sharing.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> I hope this addresses Dmitri's questions and
> > provides
> > > a
> > > > > more
> > > > > > > > > > concrete
> > > > > > > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thoughts?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Regards,
> > > > > > > > > > > > >> JB
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <
> > > > > dhuo@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Hi All,
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > One big emerging enterprise use case coming up
> as
> > > more
> > > > > > > people
> > > > > > > > > > > > >> consolidate
> > > > > > > > > > > > >> > Data Lakehouses and Catalogs is something
> commonly
> > > known
> > > > > > as
> > > > > > > > > "Data
> > > > > > > > > > > > >> Sharing",
> > > > > > > > > > > > >> > an more specifically over the course of adoption
> > of
> > > open
> > > > > > > table
> > > > > > > > > > formats
> > > > > > > > > > > > >> > "Open Data Sharing".
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Examples of existing managed service providers'
> > Data
> > > > > > Sharing
> > > > > > > > > > features:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> https://www.databricks.com/product/delta-sharing
> > > > > > > > > > > > >> >
> > > > > > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > > > > > > >> >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > > > > > > >> >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > The basic idea is that when you share data
> between
> > > > > > different
> > > > > > > > > > > > companies,
> > > > > > > > > > > > >> you
> > > > > > > > > > > > >> > need a first-class governance/management layer
> and
> > > extra
> > > > > > > > > > > > >> bells-and-whistles
> > > > > > > > > > > > >> > that are distinct from just the basic
> capabilities
> > > of
> > > > > RBAC
> > > > > > > or
> > > > > > > > > > > > >> generalized
> > > > > > > > > > > > >> > access-control (i.e. if you're sharing across
> > > > > > > > > partially-untrusted
> > > > > > > > > > org
> > > > > > > > > > > > >> > boundaries, you don't just let the consumer
> > > organization
> > > > > > log
> > > > > > > > > into
> > > > > > > > > > your
> > > > > > > > > > > > >> > datalake like one of your own employees).
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > JB and I put together this high-level proposal
> for
> > > > > > > supporting
> > > > > > > > > Open
> > > > > > > > > > > > >> Sharing
> > > > > > > > > > > > >> > in Polaris:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Tentatively, it means adding ~5 logical data
> model
> > > > > > > constructs,
> > > > > > > > > > some of
> > > > > > > > > > > > >> > which may be a first-class PolarisEntity type,
> > > others
> > > > > > > subtypes
> > > > > > > > > of
> > > > > > > > > > > > >> existing
> > > > > > > > > > > > >> > entities, and others just a nested construct:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > - ShareEntity (would behave similarly to a
> > > Catalog)
> > > > > > > > > > > > >> > - ExternalConsumer (mostly inherits from
> > > Principal)
> > > > > > > > > > > > >> > - Listing (Similar to a "role grant" but has
> > > > > different
> > > > > > > > > > metadata)
> > > > > > > > > > > > >> > - EndpointConfig (nested config under
> Listing)
> > > > > > > > > > > > >> > - ShareMembership (Similar to a "securable
> > > grant" but
> > > > > > > > > different
> > > > > > > > > > > > >> metadata)
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Feedback/comments welcome! I'll also bring it up
> > for
> > > > > live
> > > > > > > > > > discussion
> > > > > > > > > > > > if
> > > > > > > > > > > > >> > there's time in the community sync.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Cheers,
> > > > > > > > > > > > >> > Dennis
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dennis Huo <dh...@apache.org>.
Yeah, that's a good topic to discuss; I guess it touches on our core
(unresolved?) conventions on how we organize API url-prefixes and yaml
files. Reposting my response to the github comment as well:
Right, I think we could certainly split the API routes and yaml-definitions
even further if we want to ensure a well-defined auth-detail-free core, but
for the standalone-Polaris use case we'd still need to define it *somewhere*
under a Sharing-specific surface, right?
@dimas-b <https://github.com/dimas-b> Were you thinking of just splitting
the yaml file but keeping the share-management root, or adding a third
sharing prefix like
/api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials...
?
Or just yaml-file split would be keeping
/api/shares-management/v1/external-consumers/{name}/credentials... but
putting it in a separate yaml file to make it easy for service-runners to
choose not to support that whole yaml file's concrete APIs.
On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi All,
>
> Re: the shating API PR [5446], I'm repeating my GH comment here for
> visibility.
>
> I'd like the API to be neutral from the authentication perspective.
>
> This is how we handle the IRC spec in Polaris. The API declares that
> clients need to authenticate, but specific details about principals,
> clients and secrets are delegated to the server.
>
> I think this is a sound approach for the sharing API too. Specific details
> related to native Polaris principals (ConsumerCredential, etc.) can
> probably remain in the Management API.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> [5446] https://github.com/apache/polaris/pull/5446
>
> On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi
> >
> > As I said during the community meeting, let's focus on the API PR
> > first. We will build from there.
> >
> > Regards
> > JB
> >
> > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <dh...@apache.org> wrote:
> > >
> > > +1, thanks for the discussion everyone!
> > >
> > > Yes, I meant to flip it to "ready to review" earlier; flipped it now.
> I'm
> > > still working through the existing set of comments and may need to come
> > > back to it in the next few days before getting proper replies, but feel
> > > free to add additional comments.
> > >
> > > Recapping the community sync discussion re: Dmitri's last message, I'm
> > > fully aligned on building the sharing on top of existing authz
> > mechanisms,
> > > instead of being a new mechanism, and indeed external IdP is intended
> to
> > be
> > > a first-class supported use case.
> > >
> > > The latest distinction is that the ExternalConsumer isn't fundamentally
> > > identified by any internal PolarisPrincipal, but rather that it serves
> > as a
> > > wrapper that references some identity, whether that identity is defined
> > in
> > > an external IdP without any latent in-Polaris representation or if
> it's a
> > > hidden PolarisPrincipal (for the Polaris-standalone RBAC use case).
> > >
> > > The sharing endpoint via ExternalConsumer is the only layer of
> > indirection
> > > at the API entry-point, but then once the set of activePrincipalRoles
> is
> > > resolved, the underlying authz engine doesn't behave any differently
> from
> > > normal Polaris authz.
> > >
> > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > > Thanks everyone for the discussion!
> > > >
> > > > As a next step discussed in today's community sync, we will start
> > reviewing
> > > > the spec change.
> > > >
> > > > Hi Dennis, is this PR ready for review?
> > > > https://github.com/apache/polaris/pull/5446
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <
> travismbowen@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks for this discussion Dennis!
> > > > >
> > > > > Wanted to weigh in with my thoughts and questions.
> > > > >
> > > > > >
> > > > > > Do you prefer /api/shares-management/v1/
> > > > > > instead of adding to /api/management/v1?
> > > > >
> > > > >
> > > > > While I can see the argument for a share specific management
> endpoint
> > > > there
> > > > > doesn't seem to be a clear obvious choice from what I can see - I
> > lean
> > > > > towards /api/management as being the central place for all the
> > > > management -
> > > > > share or not. As long as we're careful not to leak share types or
> non
> > > > share
> > > > > types into the wrong api (which could be helped by having separate
> > yamls)
> > > > > then it feels reasonable to have all under the management api.
> > > > >
> > > > > Also went through the doc and PR and made several comments as well.
> > > > >
> > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Prithvi mentioned ExternalConsumer and principals. I'd like to
> > > > > > reinforce the need to further discuss "consumers" and principals
> > and
> > > > how
> > > > > > they relate to IdP and Polaris Authentication.
> > > > > >
> > > > > > From my POV, share access control should not be a new mechanism
> but
> > > > > should
> > > > > > build on top of the existing extensible authentication /
> > authorization
> > > > > > framework. Supporting external IdP for shares would be a valuable
> > > > feature
> > > > > > IMHO.
> > > > > >
> > > > > > Side note: I'm posting comments on PR 5446 too.
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <
> > prithvisivasankar@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Thank you for reviving this, and for the concrete spec in PR
> > 5446..
> > > > the
> > > > > > > journeys doc was especially helpful for seeing the end-to-end
> > flow.
> > > > > > >
> > > > > > > I agree this is worth doing, and that v1 is more than a
> tutorial
> > on
> > > > > > today's
> > > > > > > RBAC. A partner principal on /api/catalog is already possible;
> > what
> > > > > seems
> > > > > > > valuable is a first-class share (enumerated members, a
> restricted
> > > > > > consumer,
> > > > > > > a listing as the binding and audit unit) plus a segregated,
> > read-only
> > > > > > > Iceberg REST surface with credential vending. Stock Iceberg
> REST
> > as
> > > > the
> > > > > > > consumer protocol, and keeping Flight and Polaris-to-Polaris
> > > > federation
> > > > > > out
> > > > > > > of v1, all sound right to me.
> > > > > > >
> > > > > > > I don't have a strong view yet on the URL layout (management vs
> > > > shares
> > > > > > > prefix, nested vs top-level, prefix keyed by share vs listing)
> > and am
> > > > > > happy
> > > > > > > to follow whatever the community prefers there.
> > > > > > >
> > > > > > > A few things I wanted to check on the spec itself, several of
> > these
> > > > > echo
> > > > > > > points already raised on the list:
> > > > > > >
> > > > > > > 1. Membership as source of truth. If a hidden catalog role is
> > > > > > independently
> > > > > > > grantable, a share could be widened without going through the
> > share
> > > > > APIs.
> > > > > > > Would it make sense to say the data plane authorizes against
> > > > > membership +
> > > > > > > listing, and that any backing roles are not user-visible / not
> > > > > > > independently grantable?
> > > > > > > 2. ExternalConsumer and principals. The Aug 7 recap already
> > noted we
> > > > > > should
> > > > > > > not assume a PrincipalEntity behind every ExternalConsumer. The
> > > > > journeys
> > > > > > > still describe granting the consumer a share principal role, I
> > assume
> > > > > > that
> > > > > > > is an implementation sketch for v1 client_id/secret, not the
> > public
> > > > > > model?
> > > > > > > 3. Listing expiration. JB's original note was "share these
> tables
> > > > with
> > > > > > this
> > > > > > > consumer until this date." Credential expiry is a bit
> different.
> > > > Would
> > > > > a
> > > > > > > listing-level expiration (and maybe ACTIVE/SUSPENDED) be in
> > scope for
> > > > > v1?
> > > > > > > 4. Time-travel. As raised in the community sync, sharing via
> > Iceberg
> > > > > REST
> > > > > > > means historical snapshots remain readable. Totally fine for v1
> > to
> > > > > > support
> > > > > > > only full history, would an explicit field on the share help
> so a
> > > > later
> > > > > > > CURRENT_ONLY option is not a breaking surprise?
> > > > > > > 5. Consumer data plane. The current file is control-plane only,
> > which
> > > > > is
> > > > > > a
> > > > > > > good first step. It might help to list the Iceberg routes that
> > are
> > > > > > mounted,
> > > > > > > that writes 404, that non-members 404, and that /config does
> not
> > > > return
> > > > > > > provider catalog properties, so implementations don't diverge.
> > > > > > >
> > > > > > > Two smaller notes, if useful: in the J1 journey the consumer
> > still
> > > > > posts
> > > > > > to
> > > > > > > /api/catalog/v1/oauth/tokens, if partners are not supposed to
> > touch
> > > > the
> > > > > > > internal catalog API, should the default token endpoint live on
> > the
> > > > > > shares
> > > > > > > surface (or an external IdP)? And on Dmitri's point about share
> > > > > operators
> > > > > > > vs catalog operators, a SHARE_MANAGE privilege on the bound
> > catalog
> > > > > > > (independent of CATALOG_MANAGE_ACCESS) would make that work on
> > any
> > > > URL
> > > > > > > prefix.
> > > > > > >
> > > > > > > Happy to review the next revision, and thanks again to both of
> > you
> > > > for
> > > > > > > driving this.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Prithvi S
> > > > > > >
> > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <dh...@apache.org>
> > wrote:
> > > > > > >
> > > > > > > > As discussed live, looks like we both might have some drafts
> > we can
> > > > > > > compare
> > > > > > > > and choose/merge from :)
> > > > > > > >
> > > > > > > > Here's my current draft spec PR; it opts for my preferred
> > syntax
> > > > > > choices
> > > > > > > > for now (/api/management/ for share-management, /api/shares/
> > for
> > > > > > consumer
> > > > > > > > data-plane): https://github.com/apache/polaris/pull/5446
> > > > > > > >
> > > > > > > > Since we were talking about preferring mdfiles for design
> > iteration
> > > > > > > here's
> > > > > > > > a companion design-addendum which just enumerates what the
> > > > end-to-end
> > > > > > > user
> > > > > > > > journeys look like syntactically for each combination of
> design
> > > > > > choices:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > > > > > >
> > > > > > > > Please let me know if anyone prefers a Google Doc version of
> > the
> > > > > > > > design-addendum for easier commenting, or if the
> > design-addendum
> > > > > itself
> > > > > > > > should also be modeled as a PR on some free-floating branch
> > just to
> > > > > > > capture
> > > > > > > > comments.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <
> > > > jb@nanthrax.net
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > I am resuming work on this proposal.
> > > > > > > > >
> > > > > > > > > Regarding our recent discussion, I will move forward with
> > > > opening a
> > > > > > > > > draft PR for the /api/shares/v1 endpoint to help illustrate
> > the
> > > > > > > > > design.
> > > > > > > > >
> > > > > > > > > I will update this thread once the PR is ready.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > JB
> > > > > > > > >
> > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Dennis,
> > > > > > > > > >
> > > > > > > > > > Thanks for the recap!
> > > > > > > > > >
> > > > > > > > > > I was thinking that share management is probably distinct
> > from
> > > > > > > catalog
> > > > > > > > > > management. A user who already has a realm+catalog may
> > have an
> > > > > > option
> > > > > > > > to
> > > > > > > > > > manage shares, but not the catalog itself. Would it make
> > sense
> > > > to
> > > > > > > > > separate
> > > > > > > > > > the share management API from the general Polaris
> > management
> > > > > API? I
> > > > > > > > mean
> > > > > > > > > > using a different prefix for URIs.
> > > > > > > > > >
> > > > > > > > > > For example: /api/shares/v1/
> > > > > > > > > >
> > > > > > > > > > Obviously clients should not make any assumptions about
> the
> > > > base
> > > > > > path
> > > > > > > > > > (/api/shares), which could be different in different
> > > > deployments
> > > > > > > (e.g.
> > > > > > > > > > going through a gateway). Clients must be able to take
> the
> > base
> > > > > URI
> > > > > > > as
> > > > > > > > a
> > > > > > > > > > config option. Path segments after /v1/ will be specified
> > by
> > > > > > Polaris
> > > > > > > > > (Open
> > > > > > > > > > API yaml).
> > > > > > > > > >
> > > > > > > > > > WDYT?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <
> dhuo@apache.org
> > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Just to recap some of the discussion from today's
> > community
> > > > > sync,
> > > > > > > > > Dmitri,
> > > > > > > > > > > Robert, and Alex helped raise a few considerations to
> > roll
> > > > into
> > > > > > the
> > > > > > > > > next
> > > > > > > > > > > design/prototyping phase:
> > > > > > > > > > >
> > > > > > > > > > > 1. Given the way sharing is packaged as a separate
> > thing, it
> > > > > may
> > > > > > > not
> > > > > > > > be
> > > > > > > > > > > obvious to users that historical snapshots are equally
> > > > > available
> > > > > > to
> > > > > > > > > > > consumers as the latest snapshot (by virtue of sharing
> > via
> > > > the
> > > > > > > > > standard IRC
> > > > > > > > > > > protocol) - we should make sure to document that fact
> > well,
> > > > > > > including
> > > > > > > > > the
> > > > > > > > > > > pitfall that deleting rows by updating a table doesn't
> > > > > > necessarily
> > > > > > > > mean
> > > > > > > > > > > those rows become inaccessible to consumers
> > > > > > > > > > > 2. Make sure not to hard-code assumptions about the
> > existence
> > > > > of
> > > > > > an
> > > > > > > > > > > internal PrincipalEntity backing an ExternalConsumer -
> > even
> > > > if
> > > > > > the
> > > > > > > > > first
> > > > > > > > > > > MVP implementation uses a nested/hidden Principal of
> some
> > > > type
> > > > > > that
> > > > > > > > > holds a
> > > > > > > > > > > Polaris-managed client_id/client_secret, the 3rd-party
> > IdP
> > > > use
> > > > > > > cases
> > > > > > > > > > > require "implicit" Principals resolved via the oauth
> > token.
> > > > In
> > > > > > such
> > > > > > > > > cases
> > > > > > > > > > > the ExternalConsumer contains the metadata coordinating
> > the
> > > > > > > three-way
> > > > > > > > > > > handshake between the data-provider, data-consumer, and
> > IdP,
> > > > > but
> > > > > > > > there
> > > > > > > > > is
> > > > > > > > > > > no PrincipalEntity underlying it.
> > > > > > > > > > > 3. Longer-term, we may want to tackle "on-behalf-of"
> > > > semantics
> > > > > -
> > > > > > > > > initially,
> > > > > > > > > > > the way an ExternalConsumer works is that the
> > finer-grained
> > > > > > > consumer
> > > > > > > > > users
> > > > > > > > > > > share a single ExternalConsumer identity, just like
> how a
> > > > > > FEDERATED
> > > > > > > > > catalog
> > > > > > > > > > > shares a single service principal in its
> > ConnectionConfig.
> > > > But
> > > > > in
> > > > > > > > some
> > > > > > > > > > > cases the consumer may want some end-to-end attribution
> > of
> > > > > > > > fine-grained
> > > > > > > > > > > users, like "on-behalf-of: bob @ consumercompany.com"
> > when
> > > > the
> > > > > > > > > > > ExternalConsumer implicit principal is something like
> > > > > > > > > > > "externalconsumer$consumercompany"
> > > > > > > > > > > 4. We can iterate more rapidly on a draft spec PR and
> > also
> > > > > expand
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > > detailed technical design
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dennis
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <
> > dhuo@apache.org>
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks JB for reviving this thread, and belated
> thanks
> > to
> > > > all
> > > > > > > > > reviewers
> > > > > > > > > > > > who have left feedback in the doc or this thread so
> > far!
> > > > And
> > > > > > > > > apologies
> > > > > > > > > > > for
> > > > > > > > > > > > the silence on this, it's still been on my radar but
> I
> > got
> > > > > > > > > distracted by
> > > > > > > > > > > > some other work for awhile.
> > > > > > > > > > > >
> > > > > > > > > > > > Echoing JB's summary, the proposal is indeed that
> we're
> > > > > mostly
> > > > > > > just
> > > > > > > > > > > > reusing existing Authz/RBAC machinery, and the
> > translation
> > > > > > layer
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > new
> > > > > > > > > > > > sharing concepts sits *on top of* the existing authz
> > engine
> > > > > as
> > > > > > > much
> > > > > > > > > as
> > > > > > > > > > > > possible.
> > > > > > > > > > > >
> > > > > > > > > > > > So even if we're adding layers of indirection on
> > > > > > > > > ExternalConsumer/Listing
> > > > > > > > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the
> > parts
> > > > that
> > > > > > end
> > > > > > > > up
> > > > > > > > > > > doing
> > > > > > > > > > > > the underlying authz just reuse existing Authorizer
> > logic.
> > > > > > > > > > > >
> > > > > > > > > > > > Based on some other offline discussion with Dmitri, I
> > > > > realized
> > > > > > > why
> > > > > > > > > the
> > > > > > > > > > > > discussion around "requirements" may have been
> > confusing:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. My attempts to shorten the doc inadvertently cut
> out
> > > > some
> > > > > of
> > > > > > > the
> > > > > > > > > more
> > > > > > > > > > > > "abstract requirements" formulations
> > > > > > > > > > > > 2. The project in itself is somewhat more of a "form
> > *as*
> > > > > > > function"
> > > > > > > > > > > > feature than *strictly* functional in nature -- for
> > > > example,
> > > > > > many
> > > > > > > > > > > correctly
> > > > > > > > > > > > pointed out that "sharing" via the existing
> > RBAC/Principals
> > > > > > > syntax
> > > > > > > > is
> > > > > > > > > > > > already *possible* just by letting a cross-org
> > "consumer"
> > > > be
> > > > > a
> > > > > > > > > > > first-class
> > > > > > > > > > > > principal and access your own company's main Polaris
> > > > catalog
> > > > > > > > > endpoints.
> > > > > > > > > > > >
> > > > > > > > > > > > For (1) I added this to the Requirements section in
> > the doc
> > > > > to
> > > > > > > > better
> > > > > > > > > > > > clarify the abstract form of the data model element:
> > > > > > > > > > > >
> > > > > > > > > > > > *A. Sharing Container*: Container concept for holding
> > > > things
> > > > > > you
> > > > > > > > > want to
> > > > > > > > > > > > share - “ShareEntity”
> > > > > > > > > > > > *B. Consumer Identity*: Special flavor of
> > principal-like
> > > > > entity
> > > > > > > > > > > > representing a consumer relationship -
> > “ExternalConsumer”
> > > > > > > > > > > > *C. Map Containers to Consumers*: A way to specify
> > letting
> > > > a
> > > > > > > given
> > > > > > > > > > > > consumer access a given sharing container - “Listing”
> > > > > > > > > > > > *D. Consumer Endpoint*: A segregated API access-point
> > for
> > > > > > > > consumers -
> > > > > > > > > > > > “EndpointConfig”
> > > > > > > > > > > > *E. Objects in Containers*: A way to specify which
> > > > in-catalog
> > > > > > > > objects
> > > > > > > > > > > > belong in the big sharing container -
> “ShareMembership”
> > > > > > > > > > > >
> > > > > > > > > > > > This still bakes in a bit of implementation detail
> > since it
> > > > > > begs
> > > > > > > > the
> > > > > > > > > > > > question why not just let core RBAC/grants handle (C)
> > and
> > > > > (E).
> > > > > > To
> > > > > > > > > that
> > > > > > > > > > > end,
> > > > > > > > > > > > and addressing problem (2) of where the "form *is*
> > > > > function", I
> > > > > > > > > added a
> > > > > > > > > > > > "Concept Mapping" tab:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > > > > > >
> > > > > > > > > > > > I think the details of more advanced features can
> > still be
> > > > > > hashed
> > > > > > > > out
> > > > > > > > > > > > incrementally without blocking work on an MVP (and
> > > > certainly
> > > > > > the
> > > > > > > > MVP
> > > > > > > > > > > scope
> > > > > > > > > > > > could be trimmed so we can launch-and-iterate), so as
> > long
> > > > as
> > > > > > > > there's
> > > > > > > > > > > > community alignment on the general direction I
> propose
> > we
> > > > > could
> > > > > > > > move
> > > > > > > > > > > > forward on basic building blocks. Or if building
> blocks
> > > > > > > themselves
> > > > > > > > > are
> > > > > > > > > > > > controversial we could still proceed with some
> > prototyping
> > > > > > across
> > > > > > > > > > > different
> > > > > > > > > > > > options.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Dennis
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste
> Onofré <
> > > > > > > > > jb@nanthrax.net>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Hi everyone,
> > > > > > > > > > > >>
> > > > > > > > > > > >> I would like to resume the discussion about Data
> > Sharing
> > > > in
> > > > > > > > Polaris
> > > > > > > > > > > >> and propose focusing on three major items to help us
> > move
> > > > > > > forward.
> > > > > > > > > > > >>
> > > > > > > > > > > >> 1. What is data sharing?
> > > > > > > > > > > >> In a nutshell, it is an open protocol proposal
> > > > allowing
> > > > > > > > > > > >> organizations to share live data without copying or
> > > > > > replication.
> > > > > > > > The
> > > > > > > > > > > >> main goal is zero-copy sharing achieved through
> shared
> > > > > > metadata
> > > > > > > > and
> > > > > > > > > > > >> authorization.
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Polaris, this would mean:
> > > > > > > > > > > >> a. The consumer is authenticated as usual.
> > > > > > > > > > > >> b. The Polaris server provides the consumer with
> > direct
> > > > > access
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > >> data files (e.g., Parquet, as specified in the
> > > > > metadata.json).
> > > > > > > > > > > >> c. The consumer reads the data files directly using
> > their
> > > > > > > > preferred
> > > > > > > > > > > >> tool (e.g., a query engine).
> > > > > > > > > > > >>
> > > > > > > > > > > >> While Polaris already has most of the necessary
> > underlying
> > > > > > > > > > > >> capabilities, we need clear branding/packaging
> around
> > this
> > > > > use
> > > > > > > > case.
> > > > > > > > > > > >> Specifically, this means:
> > > > > > > > > > > >> a. A share is a dedicated catalog in the Polaris
> > server,
> > > > or
> > > > > a
> > > > > > > > > catalog
> > > > > > > > > > > >> role scoped to specific namespaces.
> > > > > > > > > > > >> b. A recipient is represented by a principal and a
> > > > principal
> > > > > > > role.
> > > > > > > > > > > >> c. A consumer profile consists of the catalog URI
> and
> > the
> > > > > > OAuth2
> > > > > > > > > > > >> client_id/client_secret.
> > > > > > > > > > > >> d. We use credential vending (temporary STS
> > credentials)
> > > > for
> > > > > > > > > consumer
> > > > > > > > > > > >> access to the data files.
> > > > > > > > > > > >> e. We use the Polaris IRC layer to expose the
> > metadata and
> > > > > the
> > > > > > > > file
> > > > > > > > > > > >> locations/access.
> > > > > > > > > > > >> f. We can also support Polaris-to-Polaris sharing
> via
> > > > > > > Federation.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Concretely:
> > > > > > > > > > > >>
> > > > > > > > > > > >> - On the provider side, we use the standard
> Polaris
> > RBAC
> > > > > > model
> > > > > > > > > > > >> (assigning grants to catalog roles, and assigning
> > those
> > > > > > catalog
> > > > > > > > > roles
> > > > > > > > > > > >> to principal roles):
> > > > > > > > > > > >> - Create a dedicated principal with its own
> > > > > > > > > > > client_id/client_secret.
> > > > > > > > > > > >> - Create a catalog role with TABLE_READ_DATA,
> > > > > > TABLE_LIST,
> > > > > > > > and
> > > > > > > > > > > >> NAMESPACE_LIST grants on the namespaces to be shared
> > (no
> > > > > write
> > > > > > > > > > > >> grants).
> > > > > > > > > > > >> - Create a principal role per destination,
> > attach
> > > > the
> > > > > > > > catalog
> > > > > > > > > > > >> role, and assign the principal.
> > > > > > > > > > > >> - The destination engine uses loadTable with
> the
> > > > > > > > > > > >> "X-Iceberg-Access-Delegation: vended-credentials"
> > header.
> > > > > > > Polaris
> > > > > > > > > then
> > > > > > > > > > > >> emits temporary, read-only storage credentials,
> > allowing
> > > > the
> > > > > > > > engine
> > > > > > > > > to
> > > > > > > > > > > >> read the data files directly from the owner's bucket
> > > > without
> > > > > > > > > accessing
> > > > > > > > > > > >> the rest of the bucket. This achieves zero-copy
> > sharing.
> > > > > > > > > > > >> - On the recipient/consumer side, no special
> > > > configuration
> > > > > > is
> > > > > > > > > > > >> needed; any IRC client can use it.
> > > > > > > > > > > >>
> > > > > > > > > > > >> 2. What is data sharing not?
> > > > > > > > > > > >> I previously considered adding an Arrow Flight
> > > > endpoint
> > > > > to
> > > > > > > > > > > >> Polaris, but I no longer think it is a good idea for
> > the
> > > > > > initial
> > > > > > > > > > > >> phase. An Arrow Flight endpoint would serve the data
> > > > > directly,
> > > > > > > > which
> > > > > > > > > > > >> would require Polaris to read the data and expose
> it.
> > I am
> > > > > > > against
> > > > > > > > > > > >> embedding a query engine in Polaris, as this tight
> > > > coupling
> > > > > > > feels
> > > > > > > > > like
> > > > > > > > > > > >> an anti-pattern. While we could use a delegation
> > service
> > > > to
> > > > > > > serve
> > > > > > > > > the
> > > > > > > > > > > >> Flight endpoint, I think it should be considered
> for a
> > > > later
> > > > > > > > phase,
> > > > > > > > > if
> > > > > > > > > > > >> at all, since a Flight endpoint might be better
> > suited as
> > > > a
> > > > > > > > consumer
> > > > > > > > > > > >> responsibility and could encourage data copying.
> > > > > > > > > > > >>
> > > > > > > > > > > >> The first phase should focus on packaging and
> > branding the
> > > > > > > > > > > >> capabilities we already have.
> > > > > > > > > > > >>
> > > > > > > > > > > >> 3. What could the Data Sharing "package" look like
> in
> > > > > > Polaris?
> > > > > > > > > > > >> To make Data Sharing obvious and easy to use, we
> > > > should
> > > > > > > > > introduce
> > > > > > > > > > > >> the missing concept of a "share"—a named definition
> > that
> > > > > users
> > > > > > > can
> > > > > > > > > > > >> list, discover, and define. The goal is to
> > materialize the
> > > > > > > user's
> > > > > > > > > > > >> intent ("I want to share these tables with this
> > consumer
> > > > > until
> > > > > > > > this
> > > > > > > > > > > >> date") into standard Polaris entities (principal,
> > > > principal
> > > > > > > role,
> > > > > > > > > > > >> catalog role, and grants).
> > > > > > > > > > > >>
> > > > > > > > > > > >> Therefore, I propose adding a /shares endpoint to
> the
> > > > > Polaris
> > > > > > > > > > > >> Management API to package and materialize these
> share
> > > > > > > definitions
> > > > > > > > > into
> > > > > > > > > > > >> Polaris entities. This would provide a usable,
> > > > first-version
> > > > > > > > package
> > > > > > > > > > > >> for Polaris Data Sharing.
> > > > > > > > > > > >>
> > > > > > > > > > > >> I hope this addresses Dmitri's questions and
> provides
> > a
> > > > more
> > > > > > > > > concrete
> > > > > > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thoughts?
> > > > > > > > > > > >>
> > > > > > > > > > > >> Regards,
> > > > > > > > > > > >> JB
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <
> > > > dhuo@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Hi All,
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > One big emerging enterprise use case coming up as
> > more
> > > > > > people
> > > > > > > > > > > >> consolidate
> > > > > > > > > > > >> > Data Lakehouses and Catalogs is something commonly
> > known
> > > > > as
> > > > > > > > "Data
> > > > > > > > > > > >> Sharing",
> > > > > > > > > > > >> > an more specifically over the course of adoption
> of
> > open
> > > > > > table
> > > > > > > > > formats
> > > > > > > > > > > >> > "Open Data Sharing".
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Examples of existing managed service providers'
> Data
> > > > > Sharing
> > > > > > > > > features:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > https://www.databricks.com/product/delta-sharing
> > > > > > > > > > > >> >
> > > > > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > > > > > >> >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > > > > > >> >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > The basic idea is that when you share data between
> > > > > different
> > > > > > > > > > > companies,
> > > > > > > > > > > >> you
> > > > > > > > > > > >> > need a first-class governance/management layer and
> > extra
> > > > > > > > > > > >> bells-and-whistles
> > > > > > > > > > > >> > that are distinct from just the basic capabilities
> > of
> > > > RBAC
> > > > > > or
> > > > > > > > > > > >> generalized
> > > > > > > > > > > >> > access-control (i.e. if you're sharing across
> > > > > > > > partially-untrusted
> > > > > > > > > org
> > > > > > > > > > > >> > boundaries, you don't just let the consumer
> > organization
> > > > > log
> > > > > > > > into
> > > > > > > > > your
> > > > > > > > > > > >> > datalake like one of your own employees).
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > JB and I put together this high-level proposal for
> > > > > > supporting
> > > > > > > > Open
> > > > > > > > > > > >> Sharing
> > > > > > > > > > > >> > in Polaris:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Tentatively, it means adding ~5 logical data model
> > > > > > constructs,
> > > > > > > > > some of
> > > > > > > > > > > >> > which may be a first-class PolarisEntity type,
> > others
> > > > > > subtypes
> > > > > > > > of
> > > > > > > > > > > >> existing
> > > > > > > > > > > >> > entities, and others just a nested construct:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > - ShareEntity (would behave similarly to a
> > Catalog)
> > > > > > > > > > > >> > - ExternalConsumer (mostly inherits from
> > Principal)
> > > > > > > > > > > >> > - Listing (Similar to a "role grant" but has
> > > > different
> > > > > > > > > metadata)
> > > > > > > > > > > >> > - EndpointConfig (nested config under Listing)
> > > > > > > > > > > >> > - ShareMembership (Similar to a "securable
> > grant" but
> > > > > > > > different
> > > > > > > > > > > >> metadata)
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Feedback/comments welcome! I'll also bring it up
> for
> > > > live
> > > > > > > > > discussion
> > > > > > > > > > > if
> > > > > > > > > > > >> > there's time in the community sync.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Cheers,
> > > > > > > > > > > >> > Dennis
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi All,
Re: the shating API PR [5446], I'm repeating my GH comment here for
visibility.
I'd like the API to be neutral from the authentication perspective.
This is how we handle the IRC spec in Polaris. The API declares that
clients need to authenticate, but specific details about principals,
clients and secrets are delegated to the server.
I think this is a sound approach for the sharing API too. Specific details
related to native Polaris principals (ConsumerCredential, etc.) can
probably remain in the Management API.
WDYT?
Thanks,
Dmitri.
[5446] https://github.com/apache/polaris/pull/5446
On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi
>
> As I said during the community meeting, let's focus on the API PR
> first. We will build from there.
>
> Regards
> JB
>
> On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <dh...@apache.org> wrote:
> >
> > +1, thanks for the discussion everyone!
> >
> > Yes, I meant to flip it to "ready to review" earlier; flipped it now. I'm
> > still working through the existing set of comments and may need to come
> > back to it in the next few days before getting proper replies, but feel
> > free to add additional comments.
> >
> > Recapping the community sync discussion re: Dmitri's last message, I'm
> > fully aligned on building the sharing on top of existing authz
> mechanisms,
> > instead of being a new mechanism, and indeed external IdP is intended to
> be
> > a first-class supported use case.
> >
> > The latest distinction is that the ExternalConsumer isn't fundamentally
> > identified by any internal PolarisPrincipal, but rather that it serves
> as a
> > wrapper that references some identity, whether that identity is defined
> in
> > an external IdP without any latent in-Polaris representation or if it's a
> > hidden PolarisPrincipal (for the Polaris-standalone RBAC use case).
> >
> > The sharing endpoint via ExternalConsumer is the only layer of
> indirection
> > at the API entry-point, but then once the set of activePrincipalRoles is
> > resolved, the underlying authz engine doesn't behave any differently from
> > normal Polaris authz.
> >
> > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Thanks everyone for the discussion!
> > >
> > > As a next step discussed in today's community sync, we will start
> reviewing
> > > the spec change.
> > >
> > > Hi Dennis, is this PR ready for review?
> > > https://github.com/apache/polaris/pull/5446
> > >
> > > Yufei
> > >
> > >
> > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <tr...@gmail.com>
> > > wrote:
> > >
> > > > Thanks for this discussion Dennis!
> > > >
> > > > Wanted to weigh in with my thoughts and questions.
> > > >
> > > > >
> > > > > Do you prefer /api/shares-management/v1/
> > > > > instead of adding to /api/management/v1?
> > > >
> > > >
> > > > While I can see the argument for a share specific management endpoint
> > > there
> > > > doesn't seem to be a clear obvious choice from what I can see - I
> lean
> > > > towards /api/management as being the central place for all the
> > > management -
> > > > share or not. As long as we're careful not to leak share types or non
> > > share
> > > > types into the wrong api (which could be helped by having separate
> yamls)
> > > > then it feels reasonable to have all under the management api.
> > > >
> > > > Also went through the doc and PR and made several comments as well.
> > > >
> > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Prithvi mentioned ExternalConsumer and principals. I'd like to
> > > > > reinforce the need to further discuss "consumers" and principals
> and
> > > how
> > > > > they relate to IdP and Polaris Authentication.
> > > > >
> > > > > From my POV, share access control should not be a new mechanism but
> > > > should
> > > > > build on top of the existing extensible authentication /
> authorization
> > > > > framework. Supporting external IdP for shares would be a valuable
> > > feature
> > > > > IMHO.
> > > > >
> > > > > Side note: I'm posting comments on PR 5446 too.
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <
> prithvisivasankar@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Thank you for reviving this, and for the concrete spec in PR
> 5446..
> > > the
> > > > > > journeys doc was especially helpful for seeing the end-to-end
> flow.
> > > > > >
> > > > > > I agree this is worth doing, and that v1 is more than a tutorial
> on
> > > > > today's
> > > > > > RBAC. A partner principal on /api/catalog is already possible;
> what
> > > > seems
> > > > > > valuable is a first-class share (enumerated members, a restricted
> > > > > consumer,
> > > > > > a listing as the binding and audit unit) plus a segregated,
> read-only
> > > > > > Iceberg REST surface with credential vending. Stock Iceberg REST
> as
> > > the
> > > > > > consumer protocol, and keeping Flight and Polaris-to-Polaris
> > > federation
> > > > > out
> > > > > > of v1, all sound right to me.
> > > > > >
> > > > > > I don't have a strong view yet on the URL layout (management vs
> > > shares
> > > > > > prefix, nested vs top-level, prefix keyed by share vs listing)
> and am
> > > > > happy
> > > > > > to follow whatever the community prefers there.
> > > > > >
> > > > > > A few things I wanted to check on the spec itself, several of
> these
> > > > echo
> > > > > > points already raised on the list:
> > > > > >
> > > > > > 1. Membership as source of truth. If a hidden catalog role is
> > > > > independently
> > > > > > grantable, a share could be widened without going through the
> share
> > > > APIs.
> > > > > > Would it make sense to say the data plane authorizes against
> > > > membership +
> > > > > > listing, and that any backing roles are not user-visible / not
> > > > > > independently grantable?
> > > > > > 2. ExternalConsumer and principals. The Aug 7 recap already
> noted we
> > > > > should
> > > > > > not assume a PrincipalEntity behind every ExternalConsumer. The
> > > > journeys
> > > > > > still describe granting the consumer a share principal role, I
> assume
> > > > > that
> > > > > > is an implementation sketch for v1 client_id/secret, not the
> public
> > > > > model?
> > > > > > 3. Listing expiration. JB's original note was "share these tables
> > > with
> > > > > this
> > > > > > consumer until this date." Credential expiry is a bit different.
> > > Would
> > > > a
> > > > > > listing-level expiration (and maybe ACTIVE/SUSPENDED) be in
> scope for
> > > > v1?
> > > > > > 4. Time-travel. As raised in the community sync, sharing via
> Iceberg
> > > > REST
> > > > > > means historical snapshots remain readable. Totally fine for v1
> to
> > > > > support
> > > > > > only full history, would an explicit field on the share help so a
> > > later
> > > > > > CURRENT_ONLY option is not a breaking surprise?
> > > > > > 5. Consumer data plane. The current file is control-plane only,
> which
> > > > is
> > > > > a
> > > > > > good first step. It might help to list the Iceberg routes that
> are
> > > > > mounted,
> > > > > > that writes 404, that non-members 404, and that /config does not
> > > return
> > > > > > provider catalog properties, so implementations don't diverge.
> > > > > >
> > > > > > Two smaller notes, if useful: in the J1 journey the consumer
> still
> > > > posts
> > > > > to
> > > > > > /api/catalog/v1/oauth/tokens, if partners are not supposed to
> touch
> > > the
> > > > > > internal catalog API, should the default token endpoint live on
> the
> > > > > shares
> > > > > > surface (or an external IdP)? And on Dmitri's point about share
> > > > operators
> > > > > > vs catalog operators, a SHARE_MANAGE privilege on the bound
> catalog
> > > > > > (independent of CATALOG_MANAGE_ACCESS) would make that work on
> any
> > > URL
> > > > > > prefix.
> > > > > >
> > > > > > Happy to review the next revision, and thanks again to both of
> you
> > > for
> > > > > > driving this.
> > > > > >
> > > > > > Regards,
> > > > > > Prithvi S
> > > > > >
> > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <dh...@apache.org>
> wrote:
> > > > > >
> > > > > > > As discussed live, looks like we both might have some drafts
> we can
> > > > > > compare
> > > > > > > and choose/merge from :)
> > > > > > >
> > > > > > > Here's my current draft spec PR; it opts for my preferred
> syntax
> > > > > choices
> > > > > > > for now (/api/management/ for share-management, /api/shares/
> for
> > > > > consumer
> > > > > > > data-plane): https://github.com/apache/polaris/pull/5446
> > > > > > >
> > > > > > > Since we were talking about preferring mdfiles for design
> iteration
> > > > > > here's
> > > > > > > a companion design-addendum which just enumerates what the
> > > end-to-end
> > > > > > user
> > > > > > > journeys look like syntactically for each combination of design
> > > > > choices:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > > > > >
> > > > > > > Please let me know if anyone prefers a Google Doc version of
> the
> > > > > > > design-addendum for easier commenting, or if the
> design-addendum
> > > > itself
> > > > > > > should also be modeled as a PR on some free-floating branch
> just to
> > > > > > capture
> > > > > > > comments.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <
> > > jb@nanthrax.net
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I am resuming work on this proposal.
> > > > > > > >
> > > > > > > > Regarding our recent discussion, I will move forward with
> > > opening a
> > > > > > > > draft PR for the /api/shares/v1 endpoint to help illustrate
> the
> > > > > > > > design.
> > > > > > > >
> > > > > > > > I will update this thread once the PR is ready.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > JB
> > > > > > > >
> > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Dennis,
> > > > > > > > >
> > > > > > > > > Thanks for the recap!
> > > > > > > > >
> > > > > > > > > I was thinking that share management is probably distinct
> from
> > > > > > catalog
> > > > > > > > > management. A user who already has a realm+catalog may
> have an
> > > > > option
> > > > > > > to
> > > > > > > > > manage shares, but not the catalog itself. Would it make
> sense
> > > to
> > > > > > > > separate
> > > > > > > > > the share management API from the general Polaris
> management
> > > > API? I
> > > > > > > mean
> > > > > > > > > using a different prefix for URIs.
> > > > > > > > >
> > > > > > > > > For example: /api/shares/v1/
> > > > > > > > >
> > > > > > > > > Obviously clients should not make any assumptions about the
> > > base
> > > > > path
> > > > > > > > > (/api/shares), which could be different in different
> > > deployments
> > > > > > (e.g.
> > > > > > > > > going through a gateway). Clients must be able to take the
> base
> > > > URI
> > > > > > as
> > > > > > > a
> > > > > > > > > config option. Path segments after /v1/ will be specified
> by
> > > > > Polaris
> > > > > > > > (Open
> > > > > > > > > API yaml).
> > > > > > > > >
> > > > > > > > > WDYT?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <dhuo@apache.org
> >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Just to recap some of the discussion from today's
> community
> > > > sync,
> > > > > > > > Dmitri,
> > > > > > > > > > Robert, and Alex helped raise a few considerations to
> roll
> > > into
> > > > > the
> > > > > > > > next
> > > > > > > > > > design/prototyping phase:
> > > > > > > > > >
> > > > > > > > > > 1. Given the way sharing is packaged as a separate
> thing, it
> > > > may
> > > > > > not
> > > > > > > be
> > > > > > > > > > obvious to users that historical snapshots are equally
> > > > available
> > > > > to
> > > > > > > > > > consumers as the latest snapshot (by virtue of sharing
> via
> > > the
> > > > > > > > standard IRC
> > > > > > > > > > protocol) - we should make sure to document that fact
> well,
> > > > > > including
> > > > > > > > the
> > > > > > > > > > pitfall that deleting rows by updating a table doesn't
> > > > > necessarily
> > > > > > > mean
> > > > > > > > > > those rows become inaccessible to consumers
> > > > > > > > > > 2. Make sure not to hard-code assumptions about the
> existence
> > > > of
> > > > > an
> > > > > > > > > > internal PrincipalEntity backing an ExternalConsumer -
> even
> > > if
> > > > > the
> > > > > > > > first
> > > > > > > > > > MVP implementation uses a nested/hidden Principal of some
> > > type
> > > > > that
> > > > > > > > holds a
> > > > > > > > > > Polaris-managed client_id/client_secret, the 3rd-party
> IdP
> > > use
> > > > > > cases
> > > > > > > > > > require "implicit" Principals resolved via the oauth
> token.
> > > In
> > > > > such
> > > > > > > > cases
> > > > > > > > > > the ExternalConsumer contains the metadata coordinating
> the
> > > > > > three-way
> > > > > > > > > > handshake between the data-provider, data-consumer, and
> IdP,
> > > > but
> > > > > > > there
> > > > > > > > is
> > > > > > > > > > no PrincipalEntity underlying it.
> > > > > > > > > > 3. Longer-term, we may want to tackle "on-behalf-of"
> > > semantics
> > > > -
> > > > > > > > initially,
> > > > > > > > > > the way an ExternalConsumer works is that the
> finer-grained
> > > > > > consumer
> > > > > > > > users
> > > > > > > > > > share a single ExternalConsumer identity, just like how a
> > > > > FEDERATED
> > > > > > > > catalog
> > > > > > > > > > shares a single service principal in its
> ConnectionConfig.
> > > But
> > > > in
> > > > > > > some
> > > > > > > > > > cases the consumer may want some end-to-end attribution
> of
> > > > > > > fine-grained
> > > > > > > > > > users, like "on-behalf-of: bob @ consumercompany.com"
> when
> > > the
> > > > > > > > > > ExternalConsumer implicit principal is something like
> > > > > > > > > > "externalconsumer$consumercompany"
> > > > > > > > > > 4. We can iterate more rapidly on a draft spec PR and
> also
> > > > expand
> > > > > > on
> > > > > > > > the
> > > > > > > > > > detailed technical design
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dennis
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <
> dhuo@apache.org>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks JB for reviving this thread, and belated thanks
> to
> > > all
> > > > > > > > reviewers
> > > > > > > > > > > who have left feedback in the doc or this thread so
> far!
> > > And
> > > > > > > > apologies
> > > > > > > > > > for
> > > > > > > > > > > the silence on this, it's still been on my radar but I
> got
> > > > > > > > distracted by
> > > > > > > > > > > some other work for awhile.
> > > > > > > > > > >
> > > > > > > > > > > Echoing JB's summary, the proposal is indeed that we're
> > > > mostly
> > > > > > just
> > > > > > > > > > > reusing existing Authz/RBAC machinery, and the
> translation
> > > > > layer
> > > > > > to
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > > sharing concepts sits *on top of* the existing authz
> engine
> > > > as
> > > > > > much
> > > > > > > > as
> > > > > > > > > > > possible.
> > > > > > > > > > >
> > > > > > > > > > > So even if we're adding layers of indirection on
> > > > > > > > ExternalConsumer/Listing
> > > > > > > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the
> parts
> > > that
> > > > > end
> > > > > > > up
> > > > > > > > > > doing
> > > > > > > > > > > the underlying authz just reuse existing Authorizer
> logic.
> > > > > > > > > > >
> > > > > > > > > > > Based on some other offline discussion with Dmitri, I
> > > > realized
> > > > > > why
> > > > > > > > the
> > > > > > > > > > > discussion around "requirements" may have been
> confusing:
> > > > > > > > > > >
> > > > > > > > > > > 1. My attempts to shorten the doc inadvertently cut out
> > > some
> > > > of
> > > > > > the
> > > > > > > > more
> > > > > > > > > > > "abstract requirements" formulations
> > > > > > > > > > > 2. The project in itself is somewhat more of a "form
> *as*
> > > > > > function"
> > > > > > > > > > > feature than *strictly* functional in nature -- for
> > > example,
> > > > > many
> > > > > > > > > > correctly
> > > > > > > > > > > pointed out that "sharing" via the existing
> RBAC/Principals
> > > > > > syntax
> > > > > > > is
> > > > > > > > > > > already *possible* just by letting a cross-org
> "consumer"
> > > be
> > > > a
> > > > > > > > > > first-class
> > > > > > > > > > > principal and access your own company's main Polaris
> > > catalog
> > > > > > > > endpoints.
> > > > > > > > > > >
> > > > > > > > > > > For (1) I added this to the Requirements section in
> the doc
> > > > to
> > > > > > > better
> > > > > > > > > > > clarify the abstract form of the data model element:
> > > > > > > > > > >
> > > > > > > > > > > *A. Sharing Container*: Container concept for holding
> > > things
> > > > > you
> > > > > > > > want to
> > > > > > > > > > > share - “ShareEntity”
> > > > > > > > > > > *B. Consumer Identity*: Special flavor of
> principal-like
> > > > entity
> > > > > > > > > > > representing a consumer relationship -
> “ExternalConsumer”
> > > > > > > > > > > *C. Map Containers to Consumers*: A way to specify
> letting
> > > a
> > > > > > given
> > > > > > > > > > > consumer access a given sharing container - “Listing”
> > > > > > > > > > > *D. Consumer Endpoint*: A segregated API access-point
> for
> > > > > > > consumers -
> > > > > > > > > > > “EndpointConfig”
> > > > > > > > > > > *E. Objects in Containers*: A way to specify which
> > > in-catalog
> > > > > > > objects
> > > > > > > > > > > belong in the big sharing container - “ShareMembership”
> > > > > > > > > > >
> > > > > > > > > > > This still bakes in a bit of implementation detail
> since it
> > > > > begs
> > > > > > > the
> > > > > > > > > > > question why not just let core RBAC/grants handle (C)
> and
> > > > (E).
> > > > > To
> > > > > > > > that
> > > > > > > > > > end,
> > > > > > > > > > > and addressing problem (2) of where the "form *is*
> > > > function", I
> > > > > > > > added a
> > > > > > > > > > > "Concept Mapping" tab:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > > > > >
> > > > > > > > > > > I think the details of more advanced features can
> still be
> > > > > hashed
> > > > > > > out
> > > > > > > > > > > incrementally without blocking work on an MVP (and
> > > certainly
> > > > > the
> > > > > > > MVP
> > > > > > > > > > scope
> > > > > > > > > > > could be trimmed so we can launch-and-iterate), so as
> long
> > > as
> > > > > > > there's
> > > > > > > > > > > community alignment on the general direction I propose
> we
> > > > could
> > > > > > > move
> > > > > > > > > > > forward on basic building blocks. Or if building blocks
> > > > > > themselves
> > > > > > > > are
> > > > > > > > > > > controversial we could still proceed with some
> prototyping
> > > > > across
> > > > > > > > > > different
> > > > > > > > > > > options.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dennis
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <
> > > > > > > > jb@nanthrax.net>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi everyone,
> > > > > > > > > > >>
> > > > > > > > > > >> I would like to resume the discussion about Data
> Sharing
> > > in
> > > > > > > Polaris
> > > > > > > > > > >> and propose focusing on three major items to help us
> move
> > > > > > forward.
> > > > > > > > > > >>
> > > > > > > > > > >> 1. What is data sharing?
> > > > > > > > > > >> In a nutshell, it is an open protocol proposal
> > > allowing
> > > > > > > > > > >> organizations to share live data without copying or
> > > > > replication.
> > > > > > > The
> > > > > > > > > > >> main goal is zero-copy sharing achieved through shared
> > > > > metadata
> > > > > > > and
> > > > > > > > > > >> authorization.
> > > > > > > > > > >>
> > > > > > > > > > >> On Polaris, this would mean:
> > > > > > > > > > >> a. The consumer is authenticated as usual.
> > > > > > > > > > >> b. The Polaris server provides the consumer with
> direct
> > > > access
> > > > > > to
> > > > > > > > the
> > > > > > > > > > >> data files (e.g., Parquet, as specified in the
> > > > metadata.json).
> > > > > > > > > > >> c. The consumer reads the data files directly using
> their
> > > > > > > preferred
> > > > > > > > > > >> tool (e.g., a query engine).
> > > > > > > > > > >>
> > > > > > > > > > >> While Polaris already has most of the necessary
> underlying
> > > > > > > > > > >> capabilities, we need clear branding/packaging around
> this
> > > > use
> > > > > > > case.
> > > > > > > > > > >> Specifically, this means:
> > > > > > > > > > >> a. A share is a dedicated catalog in the Polaris
> server,
> > > or
> > > > a
> > > > > > > > catalog
> > > > > > > > > > >> role scoped to specific namespaces.
> > > > > > > > > > >> b. A recipient is represented by a principal and a
> > > principal
> > > > > > role.
> > > > > > > > > > >> c. A consumer profile consists of the catalog URI and
> the
> > > > > OAuth2
> > > > > > > > > > >> client_id/client_secret.
> > > > > > > > > > >> d. We use credential vending (temporary STS
> credentials)
> > > for
> > > > > > > > consumer
> > > > > > > > > > >> access to the data files.
> > > > > > > > > > >> e. We use the Polaris IRC layer to expose the
> metadata and
> > > > the
> > > > > > > file
> > > > > > > > > > >> locations/access.
> > > > > > > > > > >> f. We can also support Polaris-to-Polaris sharing via
> > > > > > Federation.
> > > > > > > > > > >>
> > > > > > > > > > >> Concretely:
> > > > > > > > > > >>
> > > > > > > > > > >> - On the provider side, we use the standard Polaris
> RBAC
> > > > > model
> > > > > > > > > > >> (assigning grants to catalog roles, and assigning
> those
> > > > > catalog
> > > > > > > > roles
> > > > > > > > > > >> to principal roles):
> > > > > > > > > > >> - Create a dedicated principal with its own
> > > > > > > > > > client_id/client_secret.
> > > > > > > > > > >> - Create a catalog role with TABLE_READ_DATA,
> > > > > TABLE_LIST,
> > > > > > > and
> > > > > > > > > > >> NAMESPACE_LIST grants on the namespaces to be shared
> (no
> > > > write
> > > > > > > > > > >> grants).
> > > > > > > > > > >> - Create a principal role per destination,
> attach
> > > the
> > > > > > > catalog
> > > > > > > > > > >> role, and assign the principal.
> > > > > > > > > > >> - The destination engine uses loadTable with the
> > > > > > > > > > >> "X-Iceberg-Access-Delegation: vended-credentials"
> header.
> > > > > > Polaris
> > > > > > > > then
> > > > > > > > > > >> emits temporary, read-only storage credentials,
> allowing
> > > the
> > > > > > > engine
> > > > > > > > to
> > > > > > > > > > >> read the data files directly from the owner's bucket
> > > without
> > > > > > > > accessing
> > > > > > > > > > >> the rest of the bucket. This achieves zero-copy
> sharing.
> > > > > > > > > > >> - On the recipient/consumer side, no special
> > > configuration
> > > > > is
> > > > > > > > > > >> needed; any IRC client can use it.
> > > > > > > > > > >>
> > > > > > > > > > >> 2. What is data sharing not?
> > > > > > > > > > >> I previously considered adding an Arrow Flight
> > > endpoint
> > > > to
> > > > > > > > > > >> Polaris, but I no longer think it is a good idea for
> the
> > > > > initial
> > > > > > > > > > >> phase. An Arrow Flight endpoint would serve the data
> > > > directly,
> > > > > > > which
> > > > > > > > > > >> would require Polaris to read the data and expose it.
> I am
> > > > > > against
> > > > > > > > > > >> embedding a query engine in Polaris, as this tight
> > > coupling
> > > > > > feels
> > > > > > > > like
> > > > > > > > > > >> an anti-pattern. While we could use a delegation
> service
> > > to
> > > > > > serve
> > > > > > > > the
> > > > > > > > > > >> Flight endpoint, I think it should be considered for a
> > > later
> > > > > > > phase,
> > > > > > > > if
> > > > > > > > > > >> at all, since a Flight endpoint might be better
> suited as
> > > a
> > > > > > > consumer
> > > > > > > > > > >> responsibility and could encourage data copying.
> > > > > > > > > > >>
> > > > > > > > > > >> The first phase should focus on packaging and
> branding the
> > > > > > > > > > >> capabilities we already have.
> > > > > > > > > > >>
> > > > > > > > > > >> 3. What could the Data Sharing "package" look like in
> > > > > Polaris?
> > > > > > > > > > >> To make Data Sharing obvious and easy to use, we
> > > should
> > > > > > > > introduce
> > > > > > > > > > >> the missing concept of a "share"—a named definition
> that
> > > > users
> > > > > > can
> > > > > > > > > > >> list, discover, and define. The goal is to
> materialize the
> > > > > > user's
> > > > > > > > > > >> intent ("I want to share these tables with this
> consumer
> > > > until
> > > > > > > this
> > > > > > > > > > >> date") into standard Polaris entities (principal,
> > > principal
> > > > > > role,
> > > > > > > > > > >> catalog role, and grants).
> > > > > > > > > > >>
> > > > > > > > > > >> Therefore, I propose adding a /shares endpoint to the
> > > > Polaris
> > > > > > > > > > >> Management API to package and materialize these share
> > > > > > definitions
> > > > > > > > into
> > > > > > > > > > >> Polaris entities. This would provide a usable,
> > > first-version
> > > > > > > package
> > > > > > > > > > >> for Polaris Data Sharing.
> > > > > > > > > > >>
> > > > > > > > > > >> I hope this addresses Dmitri's questions and provides
> a
> > > more
> > > > > > > > concrete
> > > > > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > > > > >>
> > > > > > > > > > >> Thoughts?
> > > > > > > > > > >>
> > > > > > > > > > >> Regards,
> > > > > > > > > > >> JB
> > > > > > > > > > >>
> > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <
> > > dhuo@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > Hi All,
> > > > > > > > > > >> >
> > > > > > > > > > >> > One big emerging enterprise use case coming up as
> more
> > > > > people
> > > > > > > > > > >> consolidate
> > > > > > > > > > >> > Data Lakehouses and Catalogs is something commonly
> known
> > > > as
> > > > > > > "Data
> > > > > > > > > > >> Sharing",
> > > > > > > > > > >> > an more specifically over the course of adoption of
> open
> > > > > table
> > > > > > > > formats
> > > > > > > > > > >> > "Open Data Sharing".
> > > > > > > > > > >> >
> > > > > > > > > > >> > Examples of existing managed service providers' Data
> > > > Sharing
> > > > > > > > features:
> > > > > > > > > > >> >
> > > > > > > > > > >> > https://www.databricks.com/product/delta-sharing
> > > > > > > > > > >> >
> > > > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > > > > >> >
> > > > > > > > > >
> > > > > > >
> > > >
> https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > > > > >> >
> > > > > > > > > >
> > > > > > >
> > > >
> https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > > > > >> >
> > > > > > > > > > >> > The basic idea is that when you share data between
> > > > different
> > > > > > > > > > companies,
> > > > > > > > > > >> you
> > > > > > > > > > >> > need a first-class governance/management layer and
> extra
> > > > > > > > > > >> bells-and-whistles
> > > > > > > > > > >> > that are distinct from just the basic capabilities
> of
> > > RBAC
> > > > > or
> > > > > > > > > > >> generalized
> > > > > > > > > > >> > access-control (i.e. if you're sharing across
> > > > > > > partially-untrusted
> > > > > > > > org
> > > > > > > > > > >> > boundaries, you don't just let the consumer
> organization
> > > > log
> > > > > > > into
> > > > > > > > your
> > > > > > > > > > >> > datalake like one of your own employees).
> > > > > > > > > > >> >
> > > > > > > > > > >> > JB and I put together this high-level proposal for
> > > > > supporting
> > > > > > > Open
> > > > > > > > > > >> Sharing
> > > > > > > > > > >> > in Polaris:
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > > > > >> >
> > > > > > > > > > >> > Tentatively, it means adding ~5 logical data model
> > > > > constructs,
> > > > > > > > some of
> > > > > > > > > > >> > which may be a first-class PolarisEntity type,
> others
> > > > > subtypes
> > > > > > > of
> > > > > > > > > > >> existing
> > > > > > > > > > >> > entities, and others just a nested construct:
> > > > > > > > > > >> >
> > > > > > > > > > >> > - ShareEntity (would behave similarly to a
> Catalog)
> > > > > > > > > > >> > - ExternalConsumer (mostly inherits from
> Principal)
> > > > > > > > > > >> > - Listing (Similar to a "role grant" but has
> > > different
> > > > > > > > metadata)
> > > > > > > > > > >> > - EndpointConfig (nested config under Listing)
> > > > > > > > > > >> > - ShareMembership (Similar to a "securable
> grant" but
> > > > > > > different
> > > > > > > > > > >> metadata)
> > > > > > > > > > >> >
> > > > > > > > > > >> > Feedback/comments welcome! I'll also bring it up for
> > > live
> > > > > > > > discussion
> > > > > > > > > > if
> > > > > > > > > > >> > there's time in the community sync.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Cheers,
> > > > > > > > > > >> > Dennis
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi
As I said during the community meeting, let's focus on the API PR
first. We will build from there.
Regards
JB
On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <dh...@apache.org> wrote:
>
> +1, thanks for the discussion everyone!
>
> Yes, I meant to flip it to "ready to review" earlier; flipped it now. I'm
> still working through the existing set of comments and may need to come
> back to it in the next few days before getting proper replies, but feel
> free to add additional comments.
>
> Recapping the community sync discussion re: Dmitri's last message, I'm
> fully aligned on building the sharing on top of existing authz mechanisms,
> instead of being a new mechanism, and indeed external IdP is intended to be
> a first-class supported use case.
>
> The latest distinction is that the ExternalConsumer isn't fundamentally
> identified by any internal PolarisPrincipal, but rather that it serves as a
> wrapper that references some identity, whether that identity is defined in
> an external IdP without any latent in-Polaris representation or if it's a
> hidden PolarisPrincipal (for the Polaris-standalone RBAC use case).
>
> The sharing endpoint via ExternalConsumer is the only layer of indirection
> at the API entry-point, but then once the set of activePrincipalRoles is
> resolved, the underlying authz engine doesn't behave any differently from
> normal Polaris authz.
>
> On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > Thanks everyone for the discussion!
> >
> > As a next step discussed in today's community sync, we will start reviewing
> > the spec change.
> >
> > Hi Dennis, is this PR ready for review?
> > https://github.com/apache/polaris/pull/5446
> >
> > Yufei
> >
> >
> > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <tr...@gmail.com>
> > wrote:
> >
> > > Thanks for this discussion Dennis!
> > >
> > > Wanted to weigh in with my thoughts and questions.
> > >
> > > >
> > > > Do you prefer /api/shares-management/v1/
> > > > instead of adding to /api/management/v1?
> > >
> > >
> > > While I can see the argument for a share specific management endpoint
> > there
> > > doesn't seem to be a clear obvious choice from what I can see - I lean
> > > towards /api/management as being the central place for all the
> > management -
> > > share or not. As long as we're careful not to leak share types or non
> > share
> > > types into the wrong api (which could be helped by having separate yamls)
> > > then it feels reasonable to have all under the management api.
> > >
> > > Also went through the doc and PR and made several comments as well.
> > >
> > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Prithvi mentioned ExternalConsumer and principals. I'd like to
> > > > reinforce the need to further discuss "consumers" and principals and
> > how
> > > > they relate to IdP and Polaris Authentication.
> > > >
> > > > From my POV, share access control should not be a new mechanism but
> > > should
> > > > build on top of the existing extensible authentication / authorization
> > > > framework. Supporting external IdP for shares would be a valuable
> > feature
> > > > IMHO.
> > > >
> > > > Side note: I'm posting comments on PR 5446 too.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <pr...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Thank you for reviving this, and for the concrete spec in PR 5446..
> > the
> > > > > journeys doc was especially helpful for seeing the end-to-end flow.
> > > > >
> > > > > I agree this is worth doing, and that v1 is more than a tutorial on
> > > > today's
> > > > > RBAC. A partner principal on /api/catalog is already possible; what
> > > seems
> > > > > valuable is a first-class share (enumerated members, a restricted
> > > > consumer,
> > > > > a listing as the binding and audit unit) plus a segregated, read-only
> > > > > Iceberg REST surface with credential vending. Stock Iceberg REST as
> > the
> > > > > consumer protocol, and keeping Flight and Polaris-to-Polaris
> > federation
> > > > out
> > > > > of v1, all sound right to me.
> > > > >
> > > > > I don't have a strong view yet on the URL layout (management vs
> > shares
> > > > > prefix, nested vs top-level, prefix keyed by share vs listing) and am
> > > > happy
> > > > > to follow whatever the community prefers there.
> > > > >
> > > > > A few things I wanted to check on the spec itself, several of these
> > > echo
> > > > > points already raised on the list:
> > > > >
> > > > > 1. Membership as source of truth. If a hidden catalog role is
> > > > independently
> > > > > grantable, a share could be widened without going through the share
> > > APIs.
> > > > > Would it make sense to say the data plane authorizes against
> > > membership +
> > > > > listing, and that any backing roles are not user-visible / not
> > > > > independently grantable?
> > > > > 2. ExternalConsumer and principals. The Aug 7 recap already noted we
> > > > should
> > > > > not assume a PrincipalEntity behind every ExternalConsumer. The
> > > journeys
> > > > > still describe granting the consumer a share principal role, I assume
> > > > that
> > > > > is an implementation sketch for v1 client_id/secret, not the public
> > > > model?
> > > > > 3. Listing expiration. JB's original note was "share these tables
> > with
> > > > this
> > > > > consumer until this date." Credential expiry is a bit different.
> > Would
> > > a
> > > > > listing-level expiration (and maybe ACTIVE/SUSPENDED) be in scope for
> > > v1?
> > > > > 4. Time-travel. As raised in the community sync, sharing via Iceberg
> > > REST
> > > > > means historical snapshots remain readable. Totally fine for v1 to
> > > > support
> > > > > only full history, would an explicit field on the share help so a
> > later
> > > > > CURRENT_ONLY option is not a breaking surprise?
> > > > > 5. Consumer data plane. The current file is control-plane only, which
> > > is
> > > > a
> > > > > good first step. It might help to list the Iceberg routes that are
> > > > mounted,
> > > > > that writes 404, that non-members 404, and that /config does not
> > return
> > > > > provider catalog properties, so implementations don't diverge.
> > > > >
> > > > > Two smaller notes, if useful: in the J1 journey the consumer still
> > > posts
> > > > to
> > > > > /api/catalog/v1/oauth/tokens, if partners are not supposed to touch
> > the
> > > > > internal catalog API, should the default token endpoint live on the
> > > > shares
> > > > > surface (or an external IdP)? And on Dmitri's point about share
> > > operators
> > > > > vs catalog operators, a SHARE_MANAGE privilege on the bound catalog
> > > > > (independent of CATALOG_MANAGE_ACCESS) would make that work on any
> > URL
> > > > > prefix.
> > > > >
> > > > > Happy to review the next revision, and thanks again to both of you
> > for
> > > > > driving this.
> > > > >
> > > > > Regards,
> > > > > Prithvi S
> > > > >
> > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <dh...@apache.org> wrote:
> > > > >
> > > > > > As discussed live, looks like we both might have some drafts we can
> > > > > compare
> > > > > > and choose/merge from :)
> > > > > >
> > > > > > Here's my current draft spec PR; it opts for my preferred syntax
> > > > choices
> > > > > > for now (/api/management/ for share-management, /api/shares/ for
> > > > consumer
> > > > > > data-plane): https://github.com/apache/polaris/pull/5446
> > > > > >
> > > > > > Since we were talking about preferring mdfiles for design iteration
> > > > > here's
> > > > > > a companion design-addendum which just enumerates what the
> > end-to-end
> > > > > user
> > > > > > journeys look like syntactically for each combination of design
> > > > choices:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > > > >
> > > > > > Please let me know if anyone prefers a Google Doc version of the
> > > > > > design-addendum for easier commenting, or if the design-addendum
> > > itself
> > > > > > should also be modeled as a PR on some free-floating branch just to
> > > > > capture
> > > > > > comments.
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <
> > jb@nanthrax.net
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I am resuming work on this proposal.
> > > > > > >
> > > > > > > Regarding our recent discussion, I will move forward with
> > opening a
> > > > > > > draft PR for the /api/shares/v1 endpoint to help illustrate the
> > > > > > > design.
> > > > > > >
> > > > > > > I will update this thread once the PR is ready.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > JB
> > > > > > >
> > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <
> > > dimas@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Dennis,
> > > > > > > >
> > > > > > > > Thanks for the recap!
> > > > > > > >
> > > > > > > > I was thinking that share management is probably distinct from
> > > > > catalog
> > > > > > > > management. A user who already has a realm+catalog may have an
> > > > option
> > > > > > to
> > > > > > > > manage shares, but not the catalog itself. Would it make sense
> > to
> > > > > > > separate
> > > > > > > > the share management API from the general Polaris management
> > > API? I
> > > > > > mean
> > > > > > > > using a different prefix for URIs.
> > > > > > > >
> > > > > > > > For example: /api/shares/v1/
> > > > > > > >
> > > > > > > > Obviously clients should not make any assumptions about the
> > base
> > > > path
> > > > > > > > (/api/shares), which could be different in different
> > deployments
> > > > > (e.g.
> > > > > > > > going through a gateway). Clients must be able to take the base
> > > URI
> > > > > as
> > > > > > a
> > > > > > > > config option. Path segments after /v1/ will be specified by
> > > > Polaris
> > > > > > > (Open
> > > > > > > > API yaml).
> > > > > > > >
> > > > > > > > WDYT?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <dh...@apache.org>
> > > wrote:
> > > > > > > >
> > > > > > > > > Just to recap some of the discussion from today's community
> > > sync,
> > > > > > > Dmitri,
> > > > > > > > > Robert, and Alex helped raise a few considerations to roll
> > into
> > > > the
> > > > > > > next
> > > > > > > > > design/prototyping phase:
> > > > > > > > >
> > > > > > > > > 1. Given the way sharing is packaged as a separate thing, it
> > > may
> > > > > not
> > > > > > be
> > > > > > > > > obvious to users that historical snapshots are equally
> > > available
> > > > to
> > > > > > > > > consumers as the latest snapshot (by virtue of sharing via
> > the
> > > > > > > standard IRC
> > > > > > > > > protocol) - we should make sure to document that fact well,
> > > > > including
> > > > > > > the
> > > > > > > > > pitfall that deleting rows by updating a table doesn't
> > > > necessarily
> > > > > > mean
> > > > > > > > > those rows become inaccessible to consumers
> > > > > > > > > 2. Make sure not to hard-code assumptions about the existence
> > > of
> > > > an
> > > > > > > > > internal PrincipalEntity backing an ExternalConsumer - even
> > if
> > > > the
> > > > > > > first
> > > > > > > > > MVP implementation uses a nested/hidden Principal of some
> > type
> > > > that
> > > > > > > holds a
> > > > > > > > > Polaris-managed client_id/client_secret, the 3rd-party IdP
> > use
> > > > > cases
> > > > > > > > > require "implicit" Principals resolved via the oauth token.
> > In
> > > > such
> > > > > > > cases
> > > > > > > > > the ExternalConsumer contains the metadata coordinating the
> > > > > three-way
> > > > > > > > > handshake between the data-provider, data-consumer, and IdP,
> > > but
> > > > > > there
> > > > > > > is
> > > > > > > > > no PrincipalEntity underlying it.
> > > > > > > > > 3. Longer-term, we may want to tackle "on-behalf-of"
> > semantics
> > > -
> > > > > > > initially,
> > > > > > > > > the way an ExternalConsumer works is that the finer-grained
> > > > > consumer
> > > > > > > users
> > > > > > > > > share a single ExternalConsumer identity, just like how a
> > > > FEDERATED
> > > > > > > catalog
> > > > > > > > > shares a single service principal in its ConnectionConfig.
> > But
> > > in
> > > > > > some
> > > > > > > > > cases the consumer may want some end-to-end attribution of
> > > > > > fine-grained
> > > > > > > > > users, like "on-behalf-of: bob @ consumercompany.com" when
> > the
> > > > > > > > > ExternalConsumer implicit principal is something like
> > > > > > > > > "externalconsumer$consumercompany"
> > > > > > > > > 4. We can iterate more rapidly on a draft spec PR and also
> > > expand
> > > > > on
> > > > > > > the
> > > > > > > > > detailed technical design
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dennis
> > > > > > > > >
> > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <dh...@apache.org>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks JB for reviving this thread, and belated thanks to
> > all
> > > > > > > reviewers
> > > > > > > > > > who have left feedback in the doc or this thread so far!
> > And
> > > > > > > apologies
> > > > > > > > > for
> > > > > > > > > > the silence on this, it's still been on my radar but I got
> > > > > > > distracted by
> > > > > > > > > > some other work for awhile.
> > > > > > > > > >
> > > > > > > > > > Echoing JB's summary, the proposal is indeed that we're
> > > mostly
> > > > > just
> > > > > > > > > > reusing existing Authz/RBAC machinery, and the translation
> > > > layer
> > > > > to
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > sharing concepts sits *on top of* the existing authz engine
> > > as
> > > > > much
> > > > > > > as
> > > > > > > > > > possible.
> > > > > > > > > >
> > > > > > > > > > So even if we're adding layers of indirection on
> > > > > > > ExternalConsumer/Listing
> > > > > > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the parts
> > that
> > > > end
> > > > > > up
> > > > > > > > > doing
> > > > > > > > > > the underlying authz just reuse existing Authorizer logic.
> > > > > > > > > >
> > > > > > > > > > Based on some other offline discussion with Dmitri, I
> > > realized
> > > > > why
> > > > > > > the
> > > > > > > > > > discussion around "requirements" may have been confusing:
> > > > > > > > > >
> > > > > > > > > > 1. My attempts to shorten the doc inadvertently cut out
> > some
> > > of
> > > > > the
> > > > > > > more
> > > > > > > > > > "abstract requirements" formulations
> > > > > > > > > > 2. The project in itself is somewhat more of a "form *as*
> > > > > function"
> > > > > > > > > > feature than *strictly* functional in nature -- for
> > example,
> > > > many
> > > > > > > > > correctly
> > > > > > > > > > pointed out that "sharing" via the existing RBAC/Principals
> > > > > syntax
> > > > > > is
> > > > > > > > > > already *possible* just by letting a cross-org "consumer"
> > be
> > > a
> > > > > > > > > first-class
> > > > > > > > > > principal and access your own company's main Polaris
> > catalog
> > > > > > > endpoints.
> > > > > > > > > >
> > > > > > > > > > For (1) I added this to the Requirements section in the doc
> > > to
> > > > > > better
> > > > > > > > > > clarify the abstract form of the data model element:
> > > > > > > > > >
> > > > > > > > > > *A. Sharing Container*: Container concept for holding
> > things
> > > > you
> > > > > > > want to
> > > > > > > > > > share - “ShareEntity”
> > > > > > > > > > *B. Consumer Identity*: Special flavor of principal-like
> > > entity
> > > > > > > > > > representing a consumer relationship - “ExternalConsumer”
> > > > > > > > > > *C. Map Containers to Consumers*: A way to specify letting
> > a
> > > > > given
> > > > > > > > > > consumer access a given sharing container - “Listing”
> > > > > > > > > > *D. Consumer Endpoint*: A segregated API access-point for
> > > > > > consumers -
> > > > > > > > > > “EndpointConfig”
> > > > > > > > > > *E. Objects in Containers*: A way to specify which
> > in-catalog
> > > > > > objects
> > > > > > > > > > belong in the big sharing container - “ShareMembership”
> > > > > > > > > >
> > > > > > > > > > This still bakes in a bit of implementation detail since it
> > > > begs
> > > > > > the
> > > > > > > > > > question why not just let core RBAC/grants handle (C) and
> > > (E).
> > > > To
> > > > > > > that
> > > > > > > > > end,
> > > > > > > > > > and addressing problem (2) of where the "form *is*
> > > function", I
> > > > > > > added a
> > > > > > > > > > "Concept Mapping" tab:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > > > >
> > > > > > > > > > I think the details of more advanced features can still be
> > > > hashed
> > > > > > out
> > > > > > > > > > incrementally without blocking work on an MVP (and
> > certainly
> > > > the
> > > > > > MVP
> > > > > > > > > scope
> > > > > > > > > > could be trimmed so we can launch-and-iterate), so as long
> > as
> > > > > > there's
> > > > > > > > > > community alignment on the general direction I propose we
> > > could
> > > > > > move
> > > > > > > > > > forward on basic building blocks. Or if building blocks
> > > > > themselves
> > > > > > > are
> > > > > > > > > > controversial we could still proceed with some prototyping
> > > > across
> > > > > > > > > different
> > > > > > > > > > options.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dennis
> > > > > > > > > >
> > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <
> > > > > > > jb@nanthrax.net>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi everyone,
> > > > > > > > > >>
> > > > > > > > > >> I would like to resume the discussion about Data Sharing
> > in
> > > > > > Polaris
> > > > > > > > > >> and propose focusing on three major items to help us move
> > > > > forward.
> > > > > > > > > >>
> > > > > > > > > >> 1. What is data sharing?
> > > > > > > > > >> In a nutshell, it is an open protocol proposal
> > allowing
> > > > > > > > > >> organizations to share live data without copying or
> > > > replication.
> > > > > > The
> > > > > > > > > >> main goal is zero-copy sharing achieved through shared
> > > > metadata
> > > > > > and
> > > > > > > > > >> authorization.
> > > > > > > > > >>
> > > > > > > > > >> On Polaris, this would mean:
> > > > > > > > > >> a. The consumer is authenticated as usual.
> > > > > > > > > >> b. The Polaris server provides the consumer with direct
> > > access
> > > > > to
> > > > > > > the
> > > > > > > > > >> data files (e.g., Parquet, as specified in the
> > > metadata.json).
> > > > > > > > > >> c. The consumer reads the data files directly using their
> > > > > > preferred
> > > > > > > > > >> tool (e.g., a query engine).
> > > > > > > > > >>
> > > > > > > > > >> While Polaris already has most of the necessary underlying
> > > > > > > > > >> capabilities, we need clear branding/packaging around this
> > > use
> > > > > > case.
> > > > > > > > > >> Specifically, this means:
> > > > > > > > > >> a. A share is a dedicated catalog in the Polaris server,
> > or
> > > a
> > > > > > > catalog
> > > > > > > > > >> role scoped to specific namespaces.
> > > > > > > > > >> b. A recipient is represented by a principal and a
> > principal
> > > > > role.
> > > > > > > > > >> c. A consumer profile consists of the catalog URI and the
> > > > OAuth2
> > > > > > > > > >> client_id/client_secret.
> > > > > > > > > >> d. We use credential vending (temporary STS credentials)
> > for
> > > > > > > consumer
> > > > > > > > > >> access to the data files.
> > > > > > > > > >> e. We use the Polaris IRC layer to expose the metadata and
> > > the
> > > > > > file
> > > > > > > > > >> locations/access.
> > > > > > > > > >> f. We can also support Polaris-to-Polaris sharing via
> > > > > Federation.
> > > > > > > > > >>
> > > > > > > > > >> Concretely:
> > > > > > > > > >>
> > > > > > > > > >> - On the provider side, we use the standard Polaris RBAC
> > > > model
> > > > > > > > > >> (assigning grants to catalog roles, and assigning those
> > > > catalog
> > > > > > > roles
> > > > > > > > > >> to principal roles):
> > > > > > > > > >> - Create a dedicated principal with its own
> > > > > > > > > client_id/client_secret.
> > > > > > > > > >> - Create a catalog role with TABLE_READ_DATA,
> > > > TABLE_LIST,
> > > > > > and
> > > > > > > > > >> NAMESPACE_LIST grants on the namespaces to be shared (no
> > > write
> > > > > > > > > >> grants).
> > > > > > > > > >> - Create a principal role per destination, attach
> > the
> > > > > > catalog
> > > > > > > > > >> role, and assign the principal.
> > > > > > > > > >> - The destination engine uses loadTable with the
> > > > > > > > > >> "X-Iceberg-Access-Delegation: vended-credentials" header.
> > > > > Polaris
> > > > > > > then
> > > > > > > > > >> emits temporary, read-only storage credentials, allowing
> > the
> > > > > > engine
> > > > > > > to
> > > > > > > > > >> read the data files directly from the owner's bucket
> > without
> > > > > > > accessing
> > > > > > > > > >> the rest of the bucket. This achieves zero-copy sharing.
> > > > > > > > > >> - On the recipient/consumer side, no special
> > configuration
> > > > is
> > > > > > > > > >> needed; any IRC client can use it.
> > > > > > > > > >>
> > > > > > > > > >> 2. What is data sharing not?
> > > > > > > > > >> I previously considered adding an Arrow Flight
> > endpoint
> > > to
> > > > > > > > > >> Polaris, but I no longer think it is a good idea for the
> > > > initial
> > > > > > > > > >> phase. An Arrow Flight endpoint would serve the data
> > > directly,
> > > > > > which
> > > > > > > > > >> would require Polaris to read the data and expose it. I am
> > > > > against
> > > > > > > > > >> embedding a query engine in Polaris, as this tight
> > coupling
> > > > > feels
> > > > > > > like
> > > > > > > > > >> an anti-pattern. While we could use a delegation service
> > to
> > > > > serve
> > > > > > > the
> > > > > > > > > >> Flight endpoint, I think it should be considered for a
> > later
> > > > > > phase,
> > > > > > > if
> > > > > > > > > >> at all, since a Flight endpoint might be better suited as
> > a
> > > > > > consumer
> > > > > > > > > >> responsibility and could encourage data copying.
> > > > > > > > > >>
> > > > > > > > > >> The first phase should focus on packaging and branding the
> > > > > > > > > >> capabilities we already have.
> > > > > > > > > >>
> > > > > > > > > >> 3. What could the Data Sharing "package" look like in
> > > > Polaris?
> > > > > > > > > >> To make Data Sharing obvious and easy to use, we
> > should
> > > > > > > introduce
> > > > > > > > > >> the missing concept of a "share"—a named definition that
> > > users
> > > > > can
> > > > > > > > > >> list, discover, and define. The goal is to materialize the
> > > > > user's
> > > > > > > > > >> intent ("I want to share these tables with this consumer
> > > until
> > > > > > this
> > > > > > > > > >> date") into standard Polaris entities (principal,
> > principal
> > > > > role,
> > > > > > > > > >> catalog role, and grants).
> > > > > > > > > >>
> > > > > > > > > >> Therefore, I propose adding a /shares endpoint to the
> > > Polaris
> > > > > > > > > >> Management API to package and materialize these share
> > > > > definitions
> > > > > > > into
> > > > > > > > > >> Polaris entities. This would provide a usable,
> > first-version
> > > > > > package
> > > > > > > > > >> for Polaris Data Sharing.
> > > > > > > > > >>
> > > > > > > > > >> I hope this addresses Dmitri's questions and provides a
> > more
> > > > > > > concrete
> > > > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > > > >>
> > > > > > > > > >> Thoughts?
> > > > > > > > > >>
> > > > > > > > > >> Regards,
> > > > > > > > > >> JB
> > > > > > > > > >>
> > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <
> > dhuo@apache.org
> > > >
> > > > > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > Hi All,
> > > > > > > > > >> >
> > > > > > > > > >> > One big emerging enterprise use case coming up as more
> > > > people
> > > > > > > > > >> consolidate
> > > > > > > > > >> > Data Lakehouses and Catalogs is something commonly known
> > > as
> > > > > > "Data
> > > > > > > > > >> Sharing",
> > > > > > > > > >> > an more specifically over the course of adoption of open
> > > > table
> > > > > > > formats
> > > > > > > > > >> > "Open Data Sharing".
> > > > > > > > > >> >
> > > > > > > > > >> > Examples of existing managed service providers' Data
> > > Sharing
> > > > > > > features:
> > > > > > > > > >> >
> > > > > > > > > >> > https://www.databricks.com/product/delta-sharing
> > > > > > > > > >> >
> > > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > > > >> >
> > > > > > > > > >> > The basic idea is that when you share data between
> > > different
> > > > > > > > > companies,
> > > > > > > > > >> you
> > > > > > > > > >> > need a first-class governance/management layer and extra
> > > > > > > > > >> bells-and-whistles
> > > > > > > > > >> > that are distinct from just the basic capabilities of
> > RBAC
> > > > or
> > > > > > > > > >> generalized
> > > > > > > > > >> > access-control (i.e. if you're sharing across
> > > > > > partially-untrusted
> > > > > > > org
> > > > > > > > > >> > boundaries, you don't just let the consumer organization
> > > log
> > > > > > into
> > > > > > > your
> > > > > > > > > >> > datalake like one of your own employees).
> > > > > > > > > >> >
> > > > > > > > > >> > JB and I put together this high-level proposal for
> > > > supporting
> > > > > > Open
> > > > > > > > > >> Sharing
> > > > > > > > > >> > in Polaris:
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > > > >> >
> > > > > > > > > >> > Tentatively, it means adding ~5 logical data model
> > > > constructs,
> > > > > > > some of
> > > > > > > > > >> > which may be a first-class PolarisEntity type, others
> > > > subtypes
> > > > > > of
> > > > > > > > > >> existing
> > > > > > > > > >> > entities, and others just a nested construct:
> > > > > > > > > >> >
> > > > > > > > > >> > - ShareEntity (would behave similarly to a Catalog)
> > > > > > > > > >> > - ExternalConsumer (mostly inherits from Principal)
> > > > > > > > > >> > - Listing (Similar to a "role grant" but has
> > different
> > > > > > > metadata)
> > > > > > > > > >> > - EndpointConfig (nested config under Listing)
> > > > > > > > > >> > - ShareMembership (Similar to a "securable grant" but
> > > > > > different
> > > > > > > > > >> metadata)
> > > > > > > > > >> >
> > > > > > > > > >> > Feedback/comments welcome! I'll also bring it up for
> > live
> > > > > > > discussion
> > > > > > > > > if
> > > > > > > > > >> > there's time in the community sync.
> > > > > > > > > >> >
> > > > > > > > > >> > Cheers,
> > > > > > > > > >> > Dennis
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dennis Huo <dh...@apache.org>.
+1, thanks for the discussion everyone!
Yes, I meant to flip it to "ready to review" earlier; flipped it now. I'm
still working through the existing set of comments and may need to come
back to it in the next few days before getting proper replies, but feel
free to add additional comments.
Recapping the community sync discussion re: Dmitri's last message, I'm
fully aligned on building the sharing on top of existing authz mechanisms,
instead of being a new mechanism, and indeed external IdP is intended to be
a first-class supported use case.
The latest distinction is that the ExternalConsumer isn't fundamentally
identified by any internal PolarisPrincipal, but rather that it serves as a
wrapper that references some identity, whether that identity is defined in
an external IdP without any latent in-Polaris representation or if it's a
hidden PolarisPrincipal (for the Polaris-standalone RBAC use case).
The sharing endpoint via ExternalConsumer is the only layer of indirection
at the API entry-point, but then once the set of activePrincipalRoles is
resolved, the underlying authz engine doesn't behave any differently from
normal Polaris authz.
On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <fl...@gmail.com> wrote:
> Thanks everyone for the discussion!
>
> As a next step discussed in today's community sync, we will start reviewing
> the spec change.
>
> Hi Dennis, is this PR ready for review?
> https://github.com/apache/polaris/pull/5446
>
> Yufei
>
>
> On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <tr...@gmail.com>
> wrote:
>
> > Thanks for this discussion Dennis!
> >
> > Wanted to weigh in with my thoughts and questions.
> >
> > >
> > > Do you prefer /api/shares-management/v1/
> > > instead of adding to /api/management/v1?
> >
> >
> > While I can see the argument for a share specific management endpoint
> there
> > doesn't seem to be a clear obvious choice from what I can see - I lean
> > towards /api/management as being the central place for all the
> management -
> > share or not. As long as we're careful not to leak share types or non
> share
> > types into the wrong api (which could be helped by having separate yamls)
> > then it feels reasonable to have all under the management api.
> >
> > Also went through the doc and PR and made several comments as well.
> >
> > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi All,
> > >
> > > Prithvi mentioned ExternalConsumer and principals. I'd like to
> > > reinforce the need to further discuss "consumers" and principals and
> how
> > > they relate to IdP and Polaris Authentication.
> > >
> > > From my POV, share access control should not be a new mechanism but
> > should
> > > build on top of the existing extensible authentication / authorization
> > > framework. Supporting external IdP for shares would be a valuable
> feature
> > > IMHO.
> > >
> > > Side note: I'm posting comments on PR 5446 too.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <pr...@gmail.com>
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Thank you for reviving this, and for the concrete spec in PR 5446..
> the
> > > > journeys doc was especially helpful for seeing the end-to-end flow.
> > > >
> > > > I agree this is worth doing, and that v1 is more than a tutorial on
> > > today's
> > > > RBAC. A partner principal on /api/catalog is already possible; what
> > seems
> > > > valuable is a first-class share (enumerated members, a restricted
> > > consumer,
> > > > a listing as the binding and audit unit) plus a segregated, read-only
> > > > Iceberg REST surface with credential vending. Stock Iceberg REST as
> the
> > > > consumer protocol, and keeping Flight and Polaris-to-Polaris
> federation
> > > out
> > > > of v1, all sound right to me.
> > > >
> > > > I don't have a strong view yet on the URL layout (management vs
> shares
> > > > prefix, nested vs top-level, prefix keyed by share vs listing) and am
> > > happy
> > > > to follow whatever the community prefers there.
> > > >
> > > > A few things I wanted to check on the spec itself, several of these
> > echo
> > > > points already raised on the list:
> > > >
> > > > 1. Membership as source of truth. If a hidden catalog role is
> > > independently
> > > > grantable, a share could be widened without going through the share
> > APIs.
> > > > Would it make sense to say the data plane authorizes against
> > membership +
> > > > listing, and that any backing roles are not user-visible / not
> > > > independently grantable?
> > > > 2. ExternalConsumer and principals. The Aug 7 recap already noted we
> > > should
> > > > not assume a PrincipalEntity behind every ExternalConsumer. The
> > journeys
> > > > still describe granting the consumer a share principal role, I assume
> > > that
> > > > is an implementation sketch for v1 client_id/secret, not the public
> > > model?
> > > > 3. Listing expiration. JB's original note was "share these tables
> with
> > > this
> > > > consumer until this date." Credential expiry is a bit different.
> Would
> > a
> > > > listing-level expiration (and maybe ACTIVE/SUSPENDED) be in scope for
> > v1?
> > > > 4. Time-travel. As raised in the community sync, sharing via Iceberg
> > REST
> > > > means historical snapshots remain readable. Totally fine for v1 to
> > > support
> > > > only full history, would an explicit field on the share help so a
> later
> > > > CURRENT_ONLY option is not a breaking surprise?
> > > > 5. Consumer data plane. The current file is control-plane only, which
> > is
> > > a
> > > > good first step. It might help to list the Iceberg routes that are
> > > mounted,
> > > > that writes 404, that non-members 404, and that /config does not
> return
> > > > provider catalog properties, so implementations don't diverge.
> > > >
> > > > Two smaller notes, if useful: in the J1 journey the consumer still
> > posts
> > > to
> > > > /api/catalog/v1/oauth/tokens, if partners are not supposed to touch
> the
> > > > internal catalog API, should the default token endpoint live on the
> > > shares
> > > > surface (or an external IdP)? And on Dmitri's point about share
> > operators
> > > > vs catalog operators, a SHARE_MANAGE privilege on the bound catalog
> > > > (independent of CATALOG_MANAGE_ACCESS) would make that work on any
> URL
> > > > prefix.
> > > >
> > > > Happy to review the next revision, and thanks again to both of you
> for
> > > > driving this.
> > > >
> > > > Regards,
> > > > Prithvi S
> > > >
> > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <dh...@apache.org> wrote:
> > > >
> > > > > As discussed live, looks like we both might have some drafts we can
> > > > compare
> > > > > and choose/merge from :)
> > > > >
> > > > > Here's my current draft spec PR; it opts for my preferred syntax
> > > choices
> > > > > for now (/api/management/ for share-management, /api/shares/ for
> > > consumer
> > > > > data-plane): https://github.com/apache/polaris/pull/5446
> > > > >
> > > > > Since we were talking about preferring mdfiles for design iteration
> > > > here's
> > > > > a companion design-addendum which just enumerates what the
> end-to-end
> > > > user
> > > > > journeys look like syntactically for each combination of design
> > > choices:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > > >
> > > > > Please let me know if anyone prefers a Google Doc version of the
> > > > > design-addendum for easier commenting, or if the design-addendum
> > itself
> > > > > should also be modeled as a PR on some free-floating branch just to
> > > > capture
> > > > > comments.
> > > > >
> > > > >
> > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <
> jb@nanthrax.net
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I am resuming work on this proposal.
> > > > > >
> > > > > > Regarding our recent discussion, I will move forward with
> opening a
> > > > > > draft PR for the /api/shares/v1 endpoint to help illustrate the
> > > > > > design.
> > > > > >
> > > > > > I will update this thread once the PR is ready.
> > > > > >
> > > > > > Thanks,
> > > > > > JB
> > > > > >
> > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <
> > dimas@apache.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Dennis,
> > > > > > >
> > > > > > > Thanks for the recap!
> > > > > > >
> > > > > > > I was thinking that share management is probably distinct from
> > > > catalog
> > > > > > > management. A user who already has a realm+catalog may have an
> > > option
> > > > > to
> > > > > > > manage shares, but not the catalog itself. Would it make sense
> to
> > > > > > separate
> > > > > > > the share management API from the general Polaris management
> > API? I
> > > > > mean
> > > > > > > using a different prefix for URIs.
> > > > > > >
> > > > > > > For example: /api/shares/v1/
> > > > > > >
> > > > > > > Obviously clients should not make any assumptions about the
> base
> > > path
> > > > > > > (/api/shares), which could be different in different
> deployments
> > > > (e.g.
> > > > > > > going through a gateway). Clients must be able to take the base
> > URI
> > > > as
> > > > > a
> > > > > > > config option. Path segments after /v1/ will be specified by
> > > Polaris
> > > > > > (Open
> > > > > > > API yaml).
> > > > > > >
> > > > > > > WDYT?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <dh...@apache.org>
> > wrote:
> > > > > > >
> > > > > > > > Just to recap some of the discussion from today's community
> > sync,
> > > > > > Dmitri,
> > > > > > > > Robert, and Alex helped raise a few considerations to roll
> into
> > > the
> > > > > > next
> > > > > > > > design/prototyping phase:
> > > > > > > >
> > > > > > > > 1. Given the way sharing is packaged as a separate thing, it
> > may
> > > > not
> > > > > be
> > > > > > > > obvious to users that historical snapshots are equally
> > available
> > > to
> > > > > > > > consumers as the latest snapshot (by virtue of sharing via
> the
> > > > > > standard IRC
> > > > > > > > protocol) - we should make sure to document that fact well,
> > > > including
> > > > > > the
> > > > > > > > pitfall that deleting rows by updating a table doesn't
> > > necessarily
> > > > > mean
> > > > > > > > those rows become inaccessible to consumers
> > > > > > > > 2. Make sure not to hard-code assumptions about the existence
> > of
> > > an
> > > > > > > > internal PrincipalEntity backing an ExternalConsumer - even
> if
> > > the
> > > > > > first
> > > > > > > > MVP implementation uses a nested/hidden Principal of some
> type
> > > that
> > > > > > holds a
> > > > > > > > Polaris-managed client_id/client_secret, the 3rd-party IdP
> use
> > > > cases
> > > > > > > > require "implicit" Principals resolved via the oauth token.
> In
> > > such
> > > > > > cases
> > > > > > > > the ExternalConsumer contains the metadata coordinating the
> > > > three-way
> > > > > > > > handshake between the data-provider, data-consumer, and IdP,
> > but
> > > > > there
> > > > > > is
> > > > > > > > no PrincipalEntity underlying it.
> > > > > > > > 3. Longer-term, we may want to tackle "on-behalf-of"
> semantics
> > -
> > > > > > initially,
> > > > > > > > the way an ExternalConsumer works is that the finer-grained
> > > > consumer
> > > > > > users
> > > > > > > > share a single ExternalConsumer identity, just like how a
> > > FEDERATED
> > > > > > catalog
> > > > > > > > shares a single service principal in its ConnectionConfig.
> But
> > in
> > > > > some
> > > > > > > > cases the consumer may want some end-to-end attribution of
> > > > > fine-grained
> > > > > > > > users, like "on-behalf-of: bob @ consumercompany.com" when
> the
> > > > > > > > ExternalConsumer implicit principal is something like
> > > > > > > > "externalconsumer$consumercompany"
> > > > > > > > 4. We can iterate more rapidly on a draft spec PR and also
> > expand
> > > > on
> > > > > > the
> > > > > > > > detailed technical design
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dennis
> > > > > > > >
> > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <dh...@apache.org>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks JB for reviving this thread, and belated thanks to
> all
> > > > > > reviewers
> > > > > > > > > who have left feedback in the doc or this thread so far!
> And
> > > > > > apologies
> > > > > > > > for
> > > > > > > > > the silence on this, it's still been on my radar but I got
> > > > > > distracted by
> > > > > > > > > some other work for awhile.
> > > > > > > > >
> > > > > > > > > Echoing JB's summary, the proposal is indeed that we're
> > mostly
> > > > just
> > > > > > > > > reusing existing Authz/RBAC machinery, and the translation
> > > layer
> > > > to
> > > > > > the
> > > > > > > > new
> > > > > > > > > sharing concepts sits *on top of* the existing authz engine
> > as
> > > > much
> > > > > > as
> > > > > > > > > possible.
> > > > > > > > >
> > > > > > > > > So even if we're adding layers of indirection on
> > > > > > ExternalConsumer/Listing
> > > > > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the parts
> that
> > > end
> > > > > up
> > > > > > > > doing
> > > > > > > > > the underlying authz just reuse existing Authorizer logic.
> > > > > > > > >
> > > > > > > > > Based on some other offline discussion with Dmitri, I
> > realized
> > > > why
> > > > > > the
> > > > > > > > > discussion around "requirements" may have been confusing:
> > > > > > > > >
> > > > > > > > > 1. My attempts to shorten the doc inadvertently cut out
> some
> > of
> > > > the
> > > > > > more
> > > > > > > > > "abstract requirements" formulations
> > > > > > > > > 2. The project in itself is somewhat more of a "form *as*
> > > > function"
> > > > > > > > > feature than *strictly* functional in nature -- for
> example,
> > > many
> > > > > > > > correctly
> > > > > > > > > pointed out that "sharing" via the existing RBAC/Principals
> > > > syntax
> > > > > is
> > > > > > > > > already *possible* just by letting a cross-org "consumer"
> be
> > a
> > > > > > > > first-class
> > > > > > > > > principal and access your own company's main Polaris
> catalog
> > > > > > endpoints.
> > > > > > > > >
> > > > > > > > > For (1) I added this to the Requirements section in the doc
> > to
> > > > > better
> > > > > > > > > clarify the abstract form of the data model element:
> > > > > > > > >
> > > > > > > > > *A. Sharing Container*: Container concept for holding
> things
> > > you
> > > > > > want to
> > > > > > > > > share - “ShareEntity”
> > > > > > > > > *B. Consumer Identity*: Special flavor of principal-like
> > entity
> > > > > > > > > representing a consumer relationship - “ExternalConsumer”
> > > > > > > > > *C. Map Containers to Consumers*: A way to specify letting
> a
> > > > given
> > > > > > > > > consumer access a given sharing container - “Listing”
> > > > > > > > > *D. Consumer Endpoint*: A segregated API access-point for
> > > > > consumers -
> > > > > > > > > “EndpointConfig”
> > > > > > > > > *E. Objects in Containers*: A way to specify which
> in-catalog
> > > > > objects
> > > > > > > > > belong in the big sharing container - “ShareMembership”
> > > > > > > > >
> > > > > > > > > This still bakes in a bit of implementation detail since it
> > > begs
> > > > > the
> > > > > > > > > question why not just let core RBAC/grants handle (C) and
> > (E).
> > > To
> > > > > > that
> > > > > > > > end,
> > > > > > > > > and addressing problem (2) of where the "form *is*
> > function", I
> > > > > > added a
> > > > > > > > > "Concept Mapping" tab:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > > >
> > > > > > > > > I think the details of more advanced features can still be
> > > hashed
> > > > > out
> > > > > > > > > incrementally without blocking work on an MVP (and
> certainly
> > > the
> > > > > MVP
> > > > > > > > scope
> > > > > > > > > could be trimmed so we can launch-and-iterate), so as long
> as
> > > > > there's
> > > > > > > > > community alignment on the general direction I propose we
> > could
> > > > > move
> > > > > > > > > forward on basic building blocks. Or if building blocks
> > > > themselves
> > > > > > are
> > > > > > > > > controversial we could still proceed with some prototyping
> > > across
> > > > > > > > different
> > > > > > > > > options.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dennis
> > > > > > > > >
> > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <
> > > > > > jb@nanthrax.net>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi everyone,
> > > > > > > > >>
> > > > > > > > >> I would like to resume the discussion about Data Sharing
> in
> > > > > Polaris
> > > > > > > > >> and propose focusing on three major items to help us move
> > > > forward.
> > > > > > > > >>
> > > > > > > > >> 1. What is data sharing?
> > > > > > > > >> In a nutshell, it is an open protocol proposal
> allowing
> > > > > > > > >> organizations to share live data without copying or
> > > replication.
> > > > > The
> > > > > > > > >> main goal is zero-copy sharing achieved through shared
> > > metadata
> > > > > and
> > > > > > > > >> authorization.
> > > > > > > > >>
> > > > > > > > >> On Polaris, this would mean:
> > > > > > > > >> a. The consumer is authenticated as usual.
> > > > > > > > >> b. The Polaris server provides the consumer with direct
> > access
> > > > to
> > > > > > the
> > > > > > > > >> data files (e.g., Parquet, as specified in the
> > metadata.json).
> > > > > > > > >> c. The consumer reads the data files directly using their
> > > > > preferred
> > > > > > > > >> tool (e.g., a query engine).
> > > > > > > > >>
> > > > > > > > >> While Polaris already has most of the necessary underlying
> > > > > > > > >> capabilities, we need clear branding/packaging around this
> > use
> > > > > case.
> > > > > > > > >> Specifically, this means:
> > > > > > > > >> a. A share is a dedicated catalog in the Polaris server,
> or
> > a
> > > > > > catalog
> > > > > > > > >> role scoped to specific namespaces.
> > > > > > > > >> b. A recipient is represented by a principal and a
> principal
> > > > role.
> > > > > > > > >> c. A consumer profile consists of the catalog URI and the
> > > OAuth2
> > > > > > > > >> client_id/client_secret.
> > > > > > > > >> d. We use credential vending (temporary STS credentials)
> for
> > > > > > consumer
> > > > > > > > >> access to the data files.
> > > > > > > > >> e. We use the Polaris IRC layer to expose the metadata and
> > the
> > > > > file
> > > > > > > > >> locations/access.
> > > > > > > > >> f. We can also support Polaris-to-Polaris sharing via
> > > > Federation.
> > > > > > > > >>
> > > > > > > > >> Concretely:
> > > > > > > > >>
> > > > > > > > >> - On the provider side, we use the standard Polaris RBAC
> > > model
> > > > > > > > >> (assigning grants to catalog roles, and assigning those
> > > catalog
> > > > > > roles
> > > > > > > > >> to principal roles):
> > > > > > > > >> - Create a dedicated principal with its own
> > > > > > > > client_id/client_secret.
> > > > > > > > >> - Create a catalog role with TABLE_READ_DATA,
> > > TABLE_LIST,
> > > > > and
> > > > > > > > >> NAMESPACE_LIST grants on the namespaces to be shared (no
> > write
> > > > > > > > >> grants).
> > > > > > > > >> - Create a principal role per destination, attach
> the
> > > > > catalog
> > > > > > > > >> role, and assign the principal.
> > > > > > > > >> - The destination engine uses loadTable with the
> > > > > > > > >> "X-Iceberg-Access-Delegation: vended-credentials" header.
> > > > Polaris
> > > > > > then
> > > > > > > > >> emits temporary, read-only storage credentials, allowing
> the
> > > > > engine
> > > > > > to
> > > > > > > > >> read the data files directly from the owner's bucket
> without
> > > > > > accessing
> > > > > > > > >> the rest of the bucket. This achieves zero-copy sharing.
> > > > > > > > >> - On the recipient/consumer side, no special
> configuration
> > > is
> > > > > > > > >> needed; any IRC client can use it.
> > > > > > > > >>
> > > > > > > > >> 2. What is data sharing not?
> > > > > > > > >> I previously considered adding an Arrow Flight
> endpoint
> > to
> > > > > > > > >> Polaris, but I no longer think it is a good idea for the
> > > initial
> > > > > > > > >> phase. An Arrow Flight endpoint would serve the data
> > directly,
> > > > > which
> > > > > > > > >> would require Polaris to read the data and expose it. I am
> > > > against
> > > > > > > > >> embedding a query engine in Polaris, as this tight
> coupling
> > > > feels
> > > > > > like
> > > > > > > > >> an anti-pattern. While we could use a delegation service
> to
> > > > serve
> > > > > > the
> > > > > > > > >> Flight endpoint, I think it should be considered for a
> later
> > > > > phase,
> > > > > > if
> > > > > > > > >> at all, since a Flight endpoint might be better suited as
> a
> > > > > consumer
> > > > > > > > >> responsibility and could encourage data copying.
> > > > > > > > >>
> > > > > > > > >> The first phase should focus on packaging and branding the
> > > > > > > > >> capabilities we already have.
> > > > > > > > >>
> > > > > > > > >> 3. What could the Data Sharing "package" look like in
> > > Polaris?
> > > > > > > > >> To make Data Sharing obvious and easy to use, we
> should
> > > > > > introduce
> > > > > > > > >> the missing concept of a "share"—a named definition that
> > users
> > > > can
> > > > > > > > >> list, discover, and define. The goal is to materialize the
> > > > user's
> > > > > > > > >> intent ("I want to share these tables with this consumer
> > until
> > > > > this
> > > > > > > > >> date") into standard Polaris entities (principal,
> principal
> > > > role,
> > > > > > > > >> catalog role, and grants).
> > > > > > > > >>
> > > > > > > > >> Therefore, I propose adding a /shares endpoint to the
> > Polaris
> > > > > > > > >> Management API to package and materialize these share
> > > > definitions
> > > > > > into
> > > > > > > > >> Polaris entities. This would provide a usable,
> first-version
> > > > > package
> > > > > > > > >> for Polaris Data Sharing.
> > > > > > > > >>
> > > > > > > > >> I hope this addresses Dmitri's questions and provides a
> more
> > > > > > concrete
> > > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > > >>
> > > > > > > > >> Thoughts?
> > > > > > > > >>
> > > > > > > > >> Regards,
> > > > > > > > >> JB
> > > > > > > > >>
> > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <
> dhuo@apache.org
> > >
> > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > Hi All,
> > > > > > > > >> >
> > > > > > > > >> > One big emerging enterprise use case coming up as more
> > > people
> > > > > > > > >> consolidate
> > > > > > > > >> > Data Lakehouses and Catalogs is something commonly known
> > as
> > > > > "Data
> > > > > > > > >> Sharing",
> > > > > > > > >> > an more specifically over the course of adoption of open
> > > table
> > > > > > formats
> > > > > > > > >> > "Open Data Sharing".
> > > > > > > > >> >
> > > > > > > > >> > Examples of existing managed service providers' Data
> > Sharing
> > > > > > features:
> > > > > > > > >> >
> > > > > > > > >> > https://www.databricks.com/product/delta-sharing
> > > > > > > > >> >
> > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > > >> >
> > > > > > > >
> > > > >
> > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > > >> >
> > > > > > > >
> > > > >
> > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > > >> >
> > > > > > > > >> > The basic idea is that when you share data between
> > different
> > > > > > > > companies,
> > > > > > > > >> you
> > > > > > > > >> > need a first-class governance/management layer and extra
> > > > > > > > >> bells-and-whistles
> > > > > > > > >> > that are distinct from just the basic capabilities of
> RBAC
> > > or
> > > > > > > > >> generalized
> > > > > > > > >> > access-control (i.e. if you're sharing across
> > > > > partially-untrusted
> > > > > > org
> > > > > > > > >> > boundaries, you don't just let the consumer organization
> > log
> > > > > into
> > > > > > your
> > > > > > > > >> > datalake like one of your own employees).
> > > > > > > > >> >
> > > > > > > > >> > JB and I put together this high-level proposal for
> > > supporting
> > > > > Open
> > > > > > > > >> Sharing
> > > > > > > > >> > in Polaris:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > > >> >
> > > > > > > > >> > Tentatively, it means adding ~5 logical data model
> > > constructs,
> > > > > > some of
> > > > > > > > >> > which may be a first-class PolarisEntity type, others
> > > subtypes
> > > > > of
> > > > > > > > >> existing
> > > > > > > > >> > entities, and others just a nested construct:
> > > > > > > > >> >
> > > > > > > > >> > - ShareEntity (would behave similarly to a Catalog)
> > > > > > > > >> > - ExternalConsumer (mostly inherits from Principal)
> > > > > > > > >> > - Listing (Similar to a "role grant" but has
> different
> > > > > > metadata)
> > > > > > > > >> > - EndpointConfig (nested config under Listing)
> > > > > > > > >> > - ShareMembership (Similar to a "securable grant" but
> > > > > different
> > > > > > > > >> metadata)
> > > > > > > > >> >
> > > > > > > > >> > Feedback/comments welcome! I'll also bring it up for
> live
> > > > > > discussion
> > > > > > > > if
> > > > > > > > >> > there's time in the community sync.
> > > > > > > > >> >
> > > > > > > > >> > Cheers,
> > > > > > > > >> > Dennis
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Yufei Gu <fl...@gmail.com>.
Thanks everyone for the discussion!
As a next step discussed in today's community sync, we will start reviewing
the spec change.
Hi Dennis, is this PR ready for review?
https://github.com/apache/polaris/pull/5446
Yufei
On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen <tr...@gmail.com>
wrote:
> Thanks for this discussion Dennis!
>
> Wanted to weigh in with my thoughts and questions.
>
> >
> > Do you prefer /api/shares-management/v1/
> > instead of adding to /api/management/v1?
>
>
> While I can see the argument for a share specific management endpoint there
> doesn't seem to be a clear obvious choice from what I can see - I lean
> towards /api/management as being the central place for all the management -
> share or not. As long as we're careful not to leak share types or non share
> types into the wrong api (which could be helped by having separate yamls)
> then it feels reasonable to have all under the management api.
>
> Also went through the doc and PR and made several comments as well.
>
> On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > Prithvi mentioned ExternalConsumer and principals. I'd like to
> > reinforce the need to further discuss "consumers" and principals and how
> > they relate to IdP and Polaris Authentication.
> >
> > From my POV, share access control should not be a new mechanism but
> should
> > build on top of the existing extensible authentication / authorization
> > framework. Supporting external IdP for shares would be a valuable feature
> > IMHO.
> >
> > Side note: I'm posting comments on PR 5446 too.
> >
> > Cheers,
> > Dmitri.
> >
> > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <pr...@gmail.com>
> > wrote:
> >
> > > Hi everyone,
> > >
> > > Thank you for reviving this, and for the concrete spec in PR 5446.. the
> > > journeys doc was especially helpful for seeing the end-to-end flow.
> > >
> > > I agree this is worth doing, and that v1 is more than a tutorial on
> > today's
> > > RBAC. A partner principal on /api/catalog is already possible; what
> seems
> > > valuable is a first-class share (enumerated members, a restricted
> > consumer,
> > > a listing as the binding and audit unit) plus a segregated, read-only
> > > Iceberg REST surface with credential vending. Stock Iceberg REST as the
> > > consumer protocol, and keeping Flight and Polaris-to-Polaris federation
> > out
> > > of v1, all sound right to me.
> > >
> > > I don't have a strong view yet on the URL layout (management vs shares
> > > prefix, nested vs top-level, prefix keyed by share vs listing) and am
> > happy
> > > to follow whatever the community prefers there.
> > >
> > > A few things I wanted to check on the spec itself, several of these
> echo
> > > points already raised on the list:
> > >
> > > 1. Membership as source of truth. If a hidden catalog role is
> > independently
> > > grantable, a share could be widened without going through the share
> APIs.
> > > Would it make sense to say the data plane authorizes against
> membership +
> > > listing, and that any backing roles are not user-visible / not
> > > independently grantable?
> > > 2. ExternalConsumer and principals. The Aug 7 recap already noted we
> > should
> > > not assume a PrincipalEntity behind every ExternalConsumer. The
> journeys
> > > still describe granting the consumer a share principal role, I assume
> > that
> > > is an implementation sketch for v1 client_id/secret, not the public
> > model?
> > > 3. Listing expiration. JB's original note was "share these tables with
> > this
> > > consumer until this date." Credential expiry is a bit different. Would
> a
> > > listing-level expiration (and maybe ACTIVE/SUSPENDED) be in scope for
> v1?
> > > 4. Time-travel. As raised in the community sync, sharing via Iceberg
> REST
> > > means historical snapshots remain readable. Totally fine for v1 to
> > support
> > > only full history, would an explicit field on the share help so a later
> > > CURRENT_ONLY option is not a breaking surprise?
> > > 5. Consumer data plane. The current file is control-plane only, which
> is
> > a
> > > good first step. It might help to list the Iceberg routes that are
> > mounted,
> > > that writes 404, that non-members 404, and that /config does not return
> > > provider catalog properties, so implementations don't diverge.
> > >
> > > Two smaller notes, if useful: in the J1 journey the consumer still
> posts
> > to
> > > /api/catalog/v1/oauth/tokens, if partners are not supposed to touch the
> > > internal catalog API, should the default token endpoint live on the
> > shares
> > > surface (or an external IdP)? And on Dmitri's point about share
> operators
> > > vs catalog operators, a SHARE_MANAGE privilege on the bound catalog
> > > (independent of CATALOG_MANAGE_ACCESS) would make that work on any URL
> > > prefix.
> > >
> > > Happy to review the next revision, and thanks again to both of you for
> > > driving this.
> > >
> > > Regards,
> > > Prithvi S
> > >
> > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <dh...@apache.org> wrote:
> > >
> > > > As discussed live, looks like we both might have some drafts we can
> > > compare
> > > > and choose/merge from :)
> > > >
> > > > Here's my current draft spec PR; it opts for my preferred syntax
> > choices
> > > > for now (/api/management/ for share-management, /api/shares/ for
> > consumer
> > > > data-plane): https://github.com/apache/polaris/pull/5446
> > > >
> > > > Since we were talking about preferring mdfiles for design iteration
> > > here's
> > > > a companion design-addendum which just enumerates what the end-to-end
> > > user
> > > > journeys look like syntactically for each combination of design
> > choices:
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > > >
> > > > Please let me know if anyone prefers a Google Doc version of the
> > > > design-addendum for easier commenting, or if the design-addendum
> itself
> > > > should also be modeled as a PR on some free-floating branch just to
> > > capture
> > > > comments.
> > > >
> > > >
> > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <jb@nanthrax.net
> >
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I am resuming work on this proposal.
> > > > >
> > > > > Regarding our recent discussion, I will move forward with opening a
> > > > > draft PR for the /api/shares/v1 endpoint to help illustrate the
> > > > > design.
> > > > >
> > > > > I will update this thread once the PR is ready.
> > > > >
> > > > > Thanks,
> > > > > JB
> > > > >
> > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <
> dimas@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > Hi Dennis,
> > > > > >
> > > > > > Thanks for the recap!
> > > > > >
> > > > > > I was thinking that share management is probably distinct from
> > > catalog
> > > > > > management. A user who already has a realm+catalog may have an
> > option
> > > > to
> > > > > > manage shares, but not the catalog itself. Would it make sense to
> > > > > separate
> > > > > > the share management API from the general Polaris management
> API? I
> > > > mean
> > > > > > using a different prefix for URIs.
> > > > > >
> > > > > > For example: /api/shares/v1/
> > > > > >
> > > > > > Obviously clients should not make any assumptions about the base
> > path
> > > > > > (/api/shares), which could be different in different deployments
> > > (e.g.
> > > > > > going through a gateway). Clients must be able to take the base
> URI
> > > as
> > > > a
> > > > > > config option. Path segments after /v1/ will be specified by
> > Polaris
> > > > > (Open
> > > > > > API yaml).
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <dh...@apache.org>
> wrote:
> > > > > >
> > > > > > > Just to recap some of the discussion from today's community
> sync,
> > > > > Dmitri,
> > > > > > > Robert, and Alex helped raise a few considerations to roll into
> > the
> > > > > next
> > > > > > > design/prototyping phase:
> > > > > > >
> > > > > > > 1. Given the way sharing is packaged as a separate thing, it
> may
> > > not
> > > > be
> > > > > > > obvious to users that historical snapshots are equally
> available
> > to
> > > > > > > consumers as the latest snapshot (by virtue of sharing via the
> > > > > standard IRC
> > > > > > > protocol) - we should make sure to document that fact well,
> > > including
> > > > > the
> > > > > > > pitfall that deleting rows by updating a table doesn't
> > necessarily
> > > > mean
> > > > > > > those rows become inaccessible to consumers
> > > > > > > 2. Make sure not to hard-code assumptions about the existence
> of
> > an
> > > > > > > internal PrincipalEntity backing an ExternalConsumer - even if
> > the
> > > > > first
> > > > > > > MVP implementation uses a nested/hidden Principal of some type
> > that
> > > > > holds a
> > > > > > > Polaris-managed client_id/client_secret, the 3rd-party IdP use
> > > cases
> > > > > > > require "implicit" Principals resolved via the oauth token. In
> > such
> > > > > cases
> > > > > > > the ExternalConsumer contains the metadata coordinating the
> > > three-way
> > > > > > > handshake between the data-provider, data-consumer, and IdP,
> but
> > > > there
> > > > > is
> > > > > > > no PrincipalEntity underlying it.
> > > > > > > 3. Longer-term, we may want to tackle "on-behalf-of" semantics
> -
> > > > > initially,
> > > > > > > the way an ExternalConsumer works is that the finer-grained
> > > consumer
> > > > > users
> > > > > > > share a single ExternalConsumer identity, just like how a
> > FEDERATED
> > > > > catalog
> > > > > > > shares a single service principal in its ConnectionConfig. But
> in
> > > > some
> > > > > > > cases the consumer may want some end-to-end attribution of
> > > > fine-grained
> > > > > > > users, like "on-behalf-of: bob @ consumercompany.com" when the
> > > > > > > ExternalConsumer implicit principal is something like
> > > > > > > "externalconsumer$consumercompany"
> > > > > > > 4. We can iterate more rapidly on a draft spec PR and also
> expand
> > > on
> > > > > the
> > > > > > > detailed technical design
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dennis
> > > > > > >
> > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <dh...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > > > Thanks JB for reviving this thread, and belated thanks to all
> > > > > reviewers
> > > > > > > > who have left feedback in the doc or this thread so far! And
> > > > > apologies
> > > > > > > for
> > > > > > > > the silence on this, it's still been on my radar but I got
> > > > > distracted by
> > > > > > > > some other work for awhile.
> > > > > > > >
> > > > > > > > Echoing JB's summary, the proposal is indeed that we're
> mostly
> > > just
> > > > > > > > reusing existing Authz/RBAC machinery, and the translation
> > layer
> > > to
> > > > > the
> > > > > > > new
> > > > > > > > sharing concepts sits *on top of* the existing authz engine
> as
> > > much
> > > > > as
> > > > > > > > possible.
> > > > > > > >
> > > > > > > > So even if we're adding layers of indirection on
> > > > > ExternalConsumer/Listing
> > > > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the parts that
> > end
> > > > up
> > > > > > > doing
> > > > > > > > the underlying authz just reuse existing Authorizer logic.
> > > > > > > >
> > > > > > > > Based on some other offline discussion with Dmitri, I
> realized
> > > why
> > > > > the
> > > > > > > > discussion around "requirements" may have been confusing:
> > > > > > > >
> > > > > > > > 1. My attempts to shorten the doc inadvertently cut out some
> of
> > > the
> > > > > more
> > > > > > > > "abstract requirements" formulations
> > > > > > > > 2. The project in itself is somewhat more of a "form *as*
> > > function"
> > > > > > > > feature than *strictly* functional in nature -- for example,
> > many
> > > > > > > correctly
> > > > > > > > pointed out that "sharing" via the existing RBAC/Principals
> > > syntax
> > > > is
> > > > > > > > already *possible* just by letting a cross-org "consumer" be
> a
> > > > > > > first-class
> > > > > > > > principal and access your own company's main Polaris catalog
> > > > > endpoints.
> > > > > > > >
> > > > > > > > For (1) I added this to the Requirements section in the doc
> to
> > > > better
> > > > > > > > clarify the abstract form of the data model element:
> > > > > > > >
> > > > > > > > *A. Sharing Container*: Container concept for holding things
> > you
> > > > > want to
> > > > > > > > share - “ShareEntity”
> > > > > > > > *B. Consumer Identity*: Special flavor of principal-like
> entity
> > > > > > > > representing a consumer relationship - “ExternalConsumer”
> > > > > > > > *C. Map Containers to Consumers*: A way to specify letting a
> > > given
> > > > > > > > consumer access a given sharing container - “Listing”
> > > > > > > > *D. Consumer Endpoint*: A segregated API access-point for
> > > > consumers -
> > > > > > > > “EndpointConfig”
> > > > > > > > *E. Objects in Containers*: A way to specify which in-catalog
> > > > objects
> > > > > > > > belong in the big sharing container - “ShareMembership”
> > > > > > > >
> > > > > > > > This still bakes in a bit of implementation detail since it
> > begs
> > > > the
> > > > > > > > question why not just let core RBAC/grants handle (C) and
> (E).
> > To
> > > > > that
> > > > > > > end,
> > > > > > > > and addressing problem (2) of where the "form *is*
> function", I
> > > > > added a
> > > > > > > > "Concept Mapping" tab:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > > >
> > > > > > > > I think the details of more advanced features can still be
> > hashed
> > > > out
> > > > > > > > incrementally without blocking work on an MVP (and certainly
> > the
> > > > MVP
> > > > > > > scope
> > > > > > > > could be trimmed so we can launch-and-iterate), so as long as
> > > > there's
> > > > > > > > community alignment on the general direction I propose we
> could
> > > > move
> > > > > > > > forward on basic building blocks. Or if building blocks
> > > themselves
> > > > > are
> > > > > > > > controversial we could still proceed with some prototyping
> > across
> > > > > > > different
> > > > > > > > options.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dennis
> > > > > > > >
> > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <
> > > > > jb@nanthrax.net>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi everyone,
> > > > > > > >>
> > > > > > > >> I would like to resume the discussion about Data Sharing in
> > > > Polaris
> > > > > > > >> and propose focusing on three major items to help us move
> > > forward.
> > > > > > > >>
> > > > > > > >> 1. What is data sharing?
> > > > > > > >> In a nutshell, it is an open protocol proposal allowing
> > > > > > > >> organizations to share live data without copying or
> > replication.
> > > > The
> > > > > > > >> main goal is zero-copy sharing achieved through shared
> > metadata
> > > > and
> > > > > > > >> authorization.
> > > > > > > >>
> > > > > > > >> On Polaris, this would mean:
> > > > > > > >> a. The consumer is authenticated as usual.
> > > > > > > >> b. The Polaris server provides the consumer with direct
> access
> > > to
> > > > > the
> > > > > > > >> data files (e.g., Parquet, as specified in the
> metadata.json).
> > > > > > > >> c. The consumer reads the data files directly using their
> > > > preferred
> > > > > > > >> tool (e.g., a query engine).
> > > > > > > >>
> > > > > > > >> While Polaris already has most of the necessary underlying
> > > > > > > >> capabilities, we need clear branding/packaging around this
> use
> > > > case.
> > > > > > > >> Specifically, this means:
> > > > > > > >> a. A share is a dedicated catalog in the Polaris server, or
> a
> > > > > catalog
> > > > > > > >> role scoped to specific namespaces.
> > > > > > > >> b. A recipient is represented by a principal and a principal
> > > role.
> > > > > > > >> c. A consumer profile consists of the catalog URI and the
> > OAuth2
> > > > > > > >> client_id/client_secret.
> > > > > > > >> d. We use credential vending (temporary STS credentials) for
> > > > > consumer
> > > > > > > >> access to the data files.
> > > > > > > >> e. We use the Polaris IRC layer to expose the metadata and
> the
> > > > file
> > > > > > > >> locations/access.
> > > > > > > >> f. We can also support Polaris-to-Polaris sharing via
> > > Federation.
> > > > > > > >>
> > > > > > > >> Concretely:
> > > > > > > >>
> > > > > > > >> - On the provider side, we use the standard Polaris RBAC
> > model
> > > > > > > >> (assigning grants to catalog roles, and assigning those
> > catalog
> > > > > roles
> > > > > > > >> to principal roles):
> > > > > > > >> - Create a dedicated principal with its own
> > > > > > > client_id/client_secret.
> > > > > > > >> - Create a catalog role with TABLE_READ_DATA,
> > TABLE_LIST,
> > > > and
> > > > > > > >> NAMESPACE_LIST grants on the namespaces to be shared (no
> write
> > > > > > > >> grants).
> > > > > > > >> - Create a principal role per destination, attach the
> > > > catalog
> > > > > > > >> role, and assign the principal.
> > > > > > > >> - The destination engine uses loadTable with the
> > > > > > > >> "X-Iceberg-Access-Delegation: vended-credentials" header.
> > > Polaris
> > > > > then
> > > > > > > >> emits temporary, read-only storage credentials, allowing the
> > > > engine
> > > > > to
> > > > > > > >> read the data files directly from the owner's bucket without
> > > > > accessing
> > > > > > > >> the rest of the bucket. This achieves zero-copy sharing.
> > > > > > > >> - On the recipient/consumer side, no special configuration
> > is
> > > > > > > >> needed; any IRC client can use it.
> > > > > > > >>
> > > > > > > >> 2. What is data sharing not?
> > > > > > > >> I previously considered adding an Arrow Flight endpoint
> to
> > > > > > > >> Polaris, but I no longer think it is a good idea for the
> > initial
> > > > > > > >> phase. An Arrow Flight endpoint would serve the data
> directly,
> > > > which
> > > > > > > >> would require Polaris to read the data and expose it. I am
> > > against
> > > > > > > >> embedding a query engine in Polaris, as this tight coupling
> > > feels
> > > > > like
> > > > > > > >> an anti-pattern. While we could use a delegation service to
> > > serve
> > > > > the
> > > > > > > >> Flight endpoint, I think it should be considered for a later
> > > > phase,
> > > > > if
> > > > > > > >> at all, since a Flight endpoint might be better suited as a
> > > > consumer
> > > > > > > >> responsibility and could encourage data copying.
> > > > > > > >>
> > > > > > > >> The first phase should focus on packaging and branding the
> > > > > > > >> capabilities we already have.
> > > > > > > >>
> > > > > > > >> 3. What could the Data Sharing "package" look like in
> > Polaris?
> > > > > > > >> To make Data Sharing obvious and easy to use, we should
> > > > > introduce
> > > > > > > >> the missing concept of a "share"—a named definition that
> users
> > > can
> > > > > > > >> list, discover, and define. The goal is to materialize the
> > > user's
> > > > > > > >> intent ("I want to share these tables with this consumer
> until
> > > > this
> > > > > > > >> date") into standard Polaris entities (principal, principal
> > > role,
> > > > > > > >> catalog role, and grants).
> > > > > > > >>
> > > > > > > >> Therefore, I propose adding a /shares endpoint to the
> Polaris
> > > > > > > >> Management API to package and materialize these share
> > > definitions
> > > > > into
> > > > > > > >> Polaris entities. This would provide a usable, first-version
> > > > package
> > > > > > > >> for Polaris Data Sharing.
> > > > > > > >>
> > > > > > > >> I hope this addresses Dmitri's questions and provides a more
> > > > > concrete
> > > > > > > >> scenario for Polaris Data Sharing.
> > > > > > > >>
> > > > > > > >> Thoughts?
> > > > > > > >>
> > > > > > > >> Regards,
> > > > > > > >> JB
> > > > > > > >>
> > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <dhuo@apache.org
> >
> > > > wrote:
> > > > > > > >> >
> > > > > > > >> > Hi All,
> > > > > > > >> >
> > > > > > > >> > One big emerging enterprise use case coming up as more
> > people
> > > > > > > >> consolidate
> > > > > > > >> > Data Lakehouses and Catalogs is something commonly known
> as
> > > > "Data
> > > > > > > >> Sharing",
> > > > > > > >> > an more specifically over the course of adoption of open
> > table
> > > > > formats
> > > > > > > >> > "Open Data Sharing".
> > > > > > > >> >
> > > > > > > >> > Examples of existing managed service providers' Data
> Sharing
> > > > > features:
> > > > > > > >> >
> > > > > > > >> > https://www.databricks.com/product/delta-sharing
> > > > > > > >> >
> https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > > >> >
> > > > > > >
> > > >
> https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > > >> >
> > > > > > >
> > > >
> https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > > >> >
> > > > > > > >> > The basic idea is that when you share data between
> different
> > > > > > > companies,
> > > > > > > >> you
> > > > > > > >> > need a first-class governance/management layer and extra
> > > > > > > >> bells-and-whistles
> > > > > > > >> > that are distinct from just the basic capabilities of RBAC
> > or
> > > > > > > >> generalized
> > > > > > > >> > access-control (i.e. if you're sharing across
> > > > partially-untrusted
> > > > > org
> > > > > > > >> > boundaries, you don't just let the consumer organization
> log
> > > > into
> > > > > your
> > > > > > > >> > datalake like one of your own employees).
> > > > > > > >> >
> > > > > > > >> > JB and I put together this high-level proposal for
> > supporting
> > > > Open
> > > > > > > >> Sharing
> > > > > > > >> > in Polaris:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > > >> >
> > > > > > > >> > Tentatively, it means adding ~5 logical data model
> > constructs,
> > > > > some of
> > > > > > > >> > which may be a first-class PolarisEntity type, others
> > subtypes
> > > > of
> > > > > > > >> existing
> > > > > > > >> > entities, and others just a nested construct:
> > > > > > > >> >
> > > > > > > >> > - ShareEntity (would behave similarly to a Catalog)
> > > > > > > >> > - ExternalConsumer (mostly inherits from Principal)
> > > > > > > >> > - Listing (Similar to a "role grant" but has different
> > > > > metadata)
> > > > > > > >> > - EndpointConfig (nested config under Listing)
> > > > > > > >> > - ShareMembership (Similar to a "securable grant" but
> > > > different
> > > > > > > >> metadata)
> > > > > > > >> >
> > > > > > > >> > Feedback/comments welcome! I'll also bring it up for live
> > > > > discussion
> > > > > > > if
> > > > > > > >> > there's time in the community sync.
> > > > > > > >> >
> > > > > > > >> > Cheers,
> > > > > > > >> > Dennis
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Travis Bowen <tr...@gmail.com>.
Thanks for this discussion Dennis!
Wanted to weigh in with my thoughts and questions.
>
> Do you prefer /api/shares-management/v1/
> instead of adding to /api/management/v1?
While I can see the argument for a share specific management endpoint there
doesn't seem to be a clear obvious choice from what I can see - I lean
towards /api/management as being the central place for all the management -
share or not. As long as we're careful not to leak share types or non share
types into the wrong api (which could be helped by having separate yamls)
then it feels reasonable to have all under the management api.
Also went through the doc and PR and made several comments as well.
On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi All,
>
> Prithvi mentioned ExternalConsumer and principals. I'd like to
> reinforce the need to further discuss "consumers" and principals and how
> they relate to IdP and Polaris Authentication.
>
> From my POV, share access control should not be a new mechanism but should
> build on top of the existing extensible authentication / authorization
> framework. Supporting external IdP for shares would be a valuable feature
> IMHO.
>
> Side note: I'm posting comments on PR 5446 too.
>
> Cheers,
> Dmitri.
>
> On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <pr...@gmail.com>
> wrote:
>
> > Hi everyone,
> >
> > Thank you for reviving this, and for the concrete spec in PR 5446.. the
> > journeys doc was especially helpful for seeing the end-to-end flow.
> >
> > I agree this is worth doing, and that v1 is more than a tutorial on
> today's
> > RBAC. A partner principal on /api/catalog is already possible; what seems
> > valuable is a first-class share (enumerated members, a restricted
> consumer,
> > a listing as the binding and audit unit) plus a segregated, read-only
> > Iceberg REST surface with credential vending. Stock Iceberg REST as the
> > consumer protocol, and keeping Flight and Polaris-to-Polaris federation
> out
> > of v1, all sound right to me.
> >
> > I don't have a strong view yet on the URL layout (management vs shares
> > prefix, nested vs top-level, prefix keyed by share vs listing) and am
> happy
> > to follow whatever the community prefers there.
> >
> > A few things I wanted to check on the spec itself, several of these echo
> > points already raised on the list:
> >
> > 1. Membership as source of truth. If a hidden catalog role is
> independently
> > grantable, a share could be widened without going through the share APIs.
> > Would it make sense to say the data plane authorizes against membership +
> > listing, and that any backing roles are not user-visible / not
> > independently grantable?
> > 2. ExternalConsumer and principals. The Aug 7 recap already noted we
> should
> > not assume a PrincipalEntity behind every ExternalConsumer. The journeys
> > still describe granting the consumer a share principal role, I assume
> that
> > is an implementation sketch for v1 client_id/secret, not the public
> model?
> > 3. Listing expiration. JB's original note was "share these tables with
> this
> > consumer until this date." Credential expiry is a bit different. Would a
> > listing-level expiration (and maybe ACTIVE/SUSPENDED) be in scope for v1?
> > 4. Time-travel. As raised in the community sync, sharing via Iceberg REST
> > means historical snapshots remain readable. Totally fine for v1 to
> support
> > only full history, would an explicit field on the share help so a later
> > CURRENT_ONLY option is not a breaking surprise?
> > 5. Consumer data plane. The current file is control-plane only, which is
> a
> > good first step. It might help to list the Iceberg routes that are
> mounted,
> > that writes 404, that non-members 404, and that /config does not return
> > provider catalog properties, so implementations don't diverge.
> >
> > Two smaller notes, if useful: in the J1 journey the consumer still posts
> to
> > /api/catalog/v1/oauth/tokens, if partners are not supposed to touch the
> > internal catalog API, should the default token endpoint live on the
> shares
> > surface (or an external IdP)? And on Dmitri's point about share operators
> > vs catalog operators, a SHARE_MANAGE privilege on the bound catalog
> > (independent of CATALOG_MANAGE_ACCESS) would make that work on any URL
> > prefix.
> >
> > Happy to review the next revision, and thanks again to both of you for
> > driving this.
> >
> > Regards,
> > Prithvi S
> >
> > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <dh...@apache.org> wrote:
> >
> > > As discussed live, looks like we both might have some drafts we can
> > compare
> > > and choose/merge from :)
> > >
> > > Here's my current draft spec PR; it opts for my preferred syntax
> choices
> > > for now (/api/management/ for share-management, /api/shares/ for
> consumer
> > > data-plane): https://github.com/apache/polaris/pull/5446
> > >
> > > Since we were talking about preferring mdfiles for design iteration
> > here's
> > > a companion design-addendum which just enumerates what the end-to-end
> > user
> > > journeys look like syntactically for each combination of design
> choices:
> > >
> > >
> > >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> > >
> > > Please let me know if anyone prefers a Google Doc version of the
> > > design-addendum for easier commenting, or if the design-addendum itself
> > > should also be modeled as a PR on some free-floating branch just to
> > capture
> > > comments.
> > >
> > >
> > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I am resuming work on this proposal.
> > > >
> > > > Regarding our recent discussion, I will move forward with opening a
> > > > draft PR for the /api/shares/v1 endpoint to help illustrate the
> > > > design.
> > > >
> > > > I will update this thread once the PR is ready.
> > > >
> > > > Thanks,
> > > > JB
> > > >
> > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > > wrote:
> > > > >
> > > > > Hi Dennis,
> > > > >
> > > > > Thanks for the recap!
> > > > >
> > > > > I was thinking that share management is probably distinct from
> > catalog
> > > > > management. A user who already has a realm+catalog may have an
> option
> > > to
> > > > > manage shares, but not the catalog itself. Would it make sense to
> > > > separate
> > > > > the share management API from the general Polaris management API? I
> > > mean
> > > > > using a different prefix for URIs.
> > > > >
> > > > > For example: /api/shares/v1/
> > > > >
> > > > > Obviously clients should not make any assumptions about the base
> path
> > > > > (/api/shares), which could be different in different deployments
> > (e.g.
> > > > > going through a gateway). Clients must be able to take the base URI
> > as
> > > a
> > > > > config option. Path segments after /v1/ will be specified by
> Polaris
> > > > (Open
> > > > > API yaml).
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <dh...@apache.org> wrote:
> > > > >
> > > > > > Just to recap some of the discussion from today's community sync,
> > > > Dmitri,
> > > > > > Robert, and Alex helped raise a few considerations to roll into
> the
> > > > next
> > > > > > design/prototyping phase:
> > > > > >
> > > > > > 1. Given the way sharing is packaged as a separate thing, it may
> > not
> > > be
> > > > > > obvious to users that historical snapshots are equally available
> to
> > > > > > consumers as the latest snapshot (by virtue of sharing via the
> > > > standard IRC
> > > > > > protocol) - we should make sure to document that fact well,
> > including
> > > > the
> > > > > > pitfall that deleting rows by updating a table doesn't
> necessarily
> > > mean
> > > > > > those rows become inaccessible to consumers
> > > > > > 2. Make sure not to hard-code assumptions about the existence of
> an
> > > > > > internal PrincipalEntity backing an ExternalConsumer - even if
> the
> > > > first
> > > > > > MVP implementation uses a nested/hidden Principal of some type
> that
> > > > holds a
> > > > > > Polaris-managed client_id/client_secret, the 3rd-party IdP use
> > cases
> > > > > > require "implicit" Principals resolved via the oauth token. In
> such
> > > > cases
> > > > > > the ExternalConsumer contains the metadata coordinating the
> > three-way
> > > > > > handshake between the data-provider, data-consumer, and IdP, but
> > > there
> > > > is
> > > > > > no PrincipalEntity underlying it.
> > > > > > 3. Longer-term, we may want to tackle "on-behalf-of" semantics -
> > > > initially,
> > > > > > the way an ExternalConsumer works is that the finer-grained
> > consumer
> > > > users
> > > > > > share a single ExternalConsumer identity, just like how a
> FEDERATED
> > > > catalog
> > > > > > shares a single service principal in its ConnectionConfig. But in
> > > some
> > > > > > cases the consumer may want some end-to-end attribution of
> > > fine-grained
> > > > > > users, like "on-behalf-of: bob @ consumercompany.com" when the
> > > > > > ExternalConsumer implicit principal is something like
> > > > > > "externalconsumer$consumercompany"
> > > > > > 4. We can iterate more rapidly on a draft spec PR and also expand
> > on
> > > > the
> > > > > > detailed technical design
> > > > > >
> > > > > > Cheers,
> > > > > > Dennis
> > > > > >
> > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <dh...@apache.org>
> > wrote:
> > > > > >
> > > > > > > Thanks JB for reviving this thread, and belated thanks to all
> > > > reviewers
> > > > > > > who have left feedback in the doc or this thread so far! And
> > > > apologies
> > > > > > for
> > > > > > > the silence on this, it's still been on my radar but I got
> > > > distracted by
> > > > > > > some other work for awhile.
> > > > > > >
> > > > > > > Echoing JB's summary, the proposal is indeed that we're mostly
> > just
> > > > > > > reusing existing Authz/RBAC machinery, and the translation
> layer
> > to
> > > > the
> > > > > > new
> > > > > > > sharing concepts sits *on top of* the existing authz engine as
> > much
> > > > as
> > > > > > > possible.
> > > > > > >
> > > > > > > So even if we're adding layers of indirection on
> > > > ExternalConsumer/Listing
> > > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the parts that
> end
> > > up
> > > > > > doing
> > > > > > > the underlying authz just reuse existing Authorizer logic.
> > > > > > >
> > > > > > > Based on some other offline discussion with Dmitri, I realized
> > why
> > > > the
> > > > > > > discussion around "requirements" may have been confusing:
> > > > > > >
> > > > > > > 1. My attempts to shorten the doc inadvertently cut out some of
> > the
> > > > more
> > > > > > > "abstract requirements" formulations
> > > > > > > 2. The project in itself is somewhat more of a "form *as*
> > function"
> > > > > > > feature than *strictly* functional in nature -- for example,
> many
> > > > > > correctly
> > > > > > > pointed out that "sharing" via the existing RBAC/Principals
> > syntax
> > > is
> > > > > > > already *possible* just by letting a cross-org "consumer" be a
> > > > > > first-class
> > > > > > > principal and access your own company's main Polaris catalog
> > > > endpoints.
> > > > > > >
> > > > > > > For (1) I added this to the Requirements section in the doc to
> > > better
> > > > > > > clarify the abstract form of the data model element:
> > > > > > >
> > > > > > > *A. Sharing Container*: Container concept for holding things
> you
> > > > want to
> > > > > > > share - “ShareEntity”
> > > > > > > *B. Consumer Identity*: Special flavor of principal-like entity
> > > > > > > representing a consumer relationship - “ExternalConsumer”
> > > > > > > *C. Map Containers to Consumers*: A way to specify letting a
> > given
> > > > > > > consumer access a given sharing container - “Listing”
> > > > > > > *D. Consumer Endpoint*: A segregated API access-point for
> > > consumers -
> > > > > > > “EndpointConfig”
> > > > > > > *E. Objects in Containers*: A way to specify which in-catalog
> > > objects
> > > > > > > belong in the big sharing container - “ShareMembership”
> > > > > > >
> > > > > > > This still bakes in a bit of implementation detail since it
> begs
> > > the
> > > > > > > question why not just let core RBAC/grants handle (C) and (E).
> To
> > > > that
> > > > > > end,
> > > > > > > and addressing problem (2) of where the "form *is* function", I
> > > > added a
> > > > > > > "Concept Mapping" tab:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > > >
> > > > > > > I think the details of more advanced features can still be
> hashed
> > > out
> > > > > > > incrementally without blocking work on an MVP (and certainly
> the
> > > MVP
> > > > > > scope
> > > > > > > could be trimmed so we can launch-and-iterate), so as long as
> > > there's
> > > > > > > community alignment on the general direction I propose we could
> > > move
> > > > > > > forward on basic building blocks. Or if building blocks
> > themselves
> > > > are
> > > > > > > controversial we could still proceed with some prototyping
> across
> > > > > > different
> > > > > > > options.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dennis
> > > > > > >
> > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <
> > > > jb@nanthrax.net>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi everyone,
> > > > > > >>
> > > > > > >> I would like to resume the discussion about Data Sharing in
> > > Polaris
> > > > > > >> and propose focusing on three major items to help us move
> > forward.
> > > > > > >>
> > > > > > >> 1. What is data sharing?
> > > > > > >> In a nutshell, it is an open protocol proposal allowing
> > > > > > >> organizations to share live data without copying or
> replication.
> > > The
> > > > > > >> main goal is zero-copy sharing achieved through shared
> metadata
> > > and
> > > > > > >> authorization.
> > > > > > >>
> > > > > > >> On Polaris, this would mean:
> > > > > > >> a. The consumer is authenticated as usual.
> > > > > > >> b. The Polaris server provides the consumer with direct access
> > to
> > > > the
> > > > > > >> data files (e.g., Parquet, as specified in the metadata.json).
> > > > > > >> c. The consumer reads the data files directly using their
> > > preferred
> > > > > > >> tool (e.g., a query engine).
> > > > > > >>
> > > > > > >> While Polaris already has most of the necessary underlying
> > > > > > >> capabilities, we need clear branding/packaging around this use
> > > case.
> > > > > > >> Specifically, this means:
> > > > > > >> a. A share is a dedicated catalog in the Polaris server, or a
> > > > catalog
> > > > > > >> role scoped to specific namespaces.
> > > > > > >> b. A recipient is represented by a principal and a principal
> > role.
> > > > > > >> c. A consumer profile consists of the catalog URI and the
> OAuth2
> > > > > > >> client_id/client_secret.
> > > > > > >> d. We use credential vending (temporary STS credentials) for
> > > > consumer
> > > > > > >> access to the data files.
> > > > > > >> e. We use the Polaris IRC layer to expose the metadata and the
> > > file
> > > > > > >> locations/access.
> > > > > > >> f. We can also support Polaris-to-Polaris sharing via
> > Federation.
> > > > > > >>
> > > > > > >> Concretely:
> > > > > > >>
> > > > > > >> - On the provider side, we use the standard Polaris RBAC
> model
> > > > > > >> (assigning grants to catalog roles, and assigning those
> catalog
> > > > roles
> > > > > > >> to principal roles):
> > > > > > >> - Create a dedicated principal with its own
> > > > > > client_id/client_secret.
> > > > > > >> - Create a catalog role with TABLE_READ_DATA,
> TABLE_LIST,
> > > and
> > > > > > >> NAMESPACE_LIST grants on the namespaces to be shared (no write
> > > > > > >> grants).
> > > > > > >> - Create a principal role per destination, attach the
> > > catalog
> > > > > > >> role, and assign the principal.
> > > > > > >> - The destination engine uses loadTable with the
> > > > > > >> "X-Iceberg-Access-Delegation: vended-credentials" header.
> > Polaris
> > > > then
> > > > > > >> emits temporary, read-only storage credentials, allowing the
> > > engine
> > > > to
> > > > > > >> read the data files directly from the owner's bucket without
> > > > accessing
> > > > > > >> the rest of the bucket. This achieves zero-copy sharing.
> > > > > > >> - On the recipient/consumer side, no special configuration
> is
> > > > > > >> needed; any IRC client can use it.
> > > > > > >>
> > > > > > >> 2. What is data sharing not?
> > > > > > >> I previously considered adding an Arrow Flight endpoint to
> > > > > > >> Polaris, but I no longer think it is a good idea for the
> initial
> > > > > > >> phase. An Arrow Flight endpoint would serve the data directly,
> > > which
> > > > > > >> would require Polaris to read the data and expose it. I am
> > against
> > > > > > >> embedding a query engine in Polaris, as this tight coupling
> > feels
> > > > like
> > > > > > >> an anti-pattern. While we could use a delegation service to
> > serve
> > > > the
> > > > > > >> Flight endpoint, I think it should be considered for a later
> > > phase,
> > > > if
> > > > > > >> at all, since a Flight endpoint might be better suited as a
> > > consumer
> > > > > > >> responsibility and could encourage data copying.
> > > > > > >>
> > > > > > >> The first phase should focus on packaging and branding the
> > > > > > >> capabilities we already have.
> > > > > > >>
> > > > > > >> 3. What could the Data Sharing "package" look like in
> Polaris?
> > > > > > >> To make Data Sharing obvious and easy to use, we should
> > > > introduce
> > > > > > >> the missing concept of a "share"—a named definition that users
> > can
> > > > > > >> list, discover, and define. The goal is to materialize the
> > user's
> > > > > > >> intent ("I want to share these tables with this consumer until
> > > this
> > > > > > >> date") into standard Polaris entities (principal, principal
> > role,
> > > > > > >> catalog role, and grants).
> > > > > > >>
> > > > > > >> Therefore, I propose adding a /shares endpoint to the Polaris
> > > > > > >> Management API to package and materialize these share
> > definitions
> > > > into
> > > > > > >> Polaris entities. This would provide a usable, first-version
> > > package
> > > > > > >> for Polaris Data Sharing.
> > > > > > >>
> > > > > > >> I hope this addresses Dmitri's questions and provides a more
> > > > concrete
> > > > > > >> scenario for Polaris Data Sharing.
> > > > > > >>
> > > > > > >> Thoughts?
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> JB
> > > > > > >>
> > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <dh...@apache.org>
> > > wrote:
> > > > > > >> >
> > > > > > >> > Hi All,
> > > > > > >> >
> > > > > > >> > One big emerging enterprise use case coming up as more
> people
> > > > > > >> consolidate
> > > > > > >> > Data Lakehouses and Catalogs is something commonly known as
> > > "Data
> > > > > > >> Sharing",
> > > > > > >> > an more specifically over the course of adoption of open
> table
> > > > formats
> > > > > > >> > "Open Data Sharing".
> > > > > > >> >
> > > > > > >> > Examples of existing managed service providers' Data Sharing
> > > > features:
> > > > > > >> >
> > > > > > >> > https://www.databricks.com/product/delta-sharing
> > > > > > >> > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > > >> >
> > > > > >
> > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > > >> >
> > > > > >
> > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > > >> >
> > > > > > >> > The basic idea is that when you share data between different
> > > > > > companies,
> > > > > > >> you
> > > > > > >> > need a first-class governance/management layer and extra
> > > > > > >> bells-and-whistles
> > > > > > >> > that are distinct from just the basic capabilities of RBAC
> or
> > > > > > >> generalized
> > > > > > >> > access-control (i.e. if you're sharing across
> > > partially-untrusted
> > > > org
> > > > > > >> > boundaries, you don't just let the consumer organization log
> > > into
> > > > your
> > > > > > >> > datalake like one of your own employees).
> > > > > > >> >
> > > > > > >> > JB and I put together this high-level proposal for
> supporting
> > > Open
> > > > > > >> Sharing
> > > > > > >> > in Polaris:
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > > >> >
> > > > > > >> > Tentatively, it means adding ~5 logical data model
> constructs,
> > > > some of
> > > > > > >> > which may be a first-class PolarisEntity type, others
> subtypes
> > > of
> > > > > > >> existing
> > > > > > >> > entities, and others just a nested construct:
> > > > > > >> >
> > > > > > >> > - ShareEntity (would behave similarly to a Catalog)
> > > > > > >> > - ExternalConsumer (mostly inherits from Principal)
> > > > > > >> > - Listing (Similar to a "role grant" but has different
> > > > metadata)
> > > > > > >> > - EndpointConfig (nested config under Listing)
> > > > > > >> > - ShareMembership (Similar to a "securable grant" but
> > > different
> > > > > > >> metadata)
> > > > > > >> >
> > > > > > >> > Feedback/comments welcome! I'll also bring it up for live
> > > > discussion
> > > > > > if
> > > > > > >> > there's time in the community sync.
> > > > > > >> >
> > > > > > >> > Cheers,
> > > > > > >> > Dennis
> > > > > > >>
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi All,
Prithvi mentioned ExternalConsumer and principals. I'd like to
reinforce the need to further discuss "consumers" and principals and how
they relate to IdP and Polaris Authentication.
From my POV, share access control should not be a new mechanism but should
build on top of the existing extensible authentication / authorization
framework. Supporting external IdP for shares would be a valuable feature
IMHO.
Side note: I'm posting comments on PR 5446 too.
Cheers,
Dmitri.
On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <pr...@gmail.com>
wrote:
> Hi everyone,
>
> Thank you for reviving this, and for the concrete spec in PR 5446.. the
> journeys doc was especially helpful for seeing the end-to-end flow.
>
> I agree this is worth doing, and that v1 is more than a tutorial on today's
> RBAC. A partner principal on /api/catalog is already possible; what seems
> valuable is a first-class share (enumerated members, a restricted consumer,
> a listing as the binding and audit unit) plus a segregated, read-only
> Iceberg REST surface with credential vending. Stock Iceberg REST as the
> consumer protocol, and keeping Flight and Polaris-to-Polaris federation out
> of v1, all sound right to me.
>
> I don't have a strong view yet on the URL layout (management vs shares
> prefix, nested vs top-level, prefix keyed by share vs listing) and am happy
> to follow whatever the community prefers there.
>
> A few things I wanted to check on the spec itself, several of these echo
> points already raised on the list:
>
> 1. Membership as source of truth. If a hidden catalog role is independently
> grantable, a share could be widened without going through the share APIs.
> Would it make sense to say the data plane authorizes against membership +
> listing, and that any backing roles are not user-visible / not
> independently grantable?
> 2. ExternalConsumer and principals. The Aug 7 recap already noted we should
> not assume a PrincipalEntity behind every ExternalConsumer. The journeys
> still describe granting the consumer a share principal role, I assume that
> is an implementation sketch for v1 client_id/secret, not the public model?
> 3. Listing expiration. JB's original note was "share these tables with this
> consumer until this date." Credential expiry is a bit different. Would a
> listing-level expiration (and maybe ACTIVE/SUSPENDED) be in scope for v1?
> 4. Time-travel. As raised in the community sync, sharing via Iceberg REST
> means historical snapshots remain readable. Totally fine for v1 to support
> only full history, would an explicit field on the share help so a later
> CURRENT_ONLY option is not a breaking surprise?
> 5. Consumer data plane. The current file is control-plane only, which is a
> good first step. It might help to list the Iceberg routes that are mounted,
> that writes 404, that non-members 404, and that /config does not return
> provider catalog properties, so implementations don't diverge.
>
> Two smaller notes, if useful: in the J1 journey the consumer still posts to
> /api/catalog/v1/oauth/tokens, if partners are not supposed to touch the
> internal catalog API, should the default token endpoint live on the shares
> surface (or an external IdP)? And on Dmitri's point about share operators
> vs catalog operators, a SHARE_MANAGE privilege on the bound catalog
> (independent of CATALOG_MANAGE_ACCESS) would make that work on any URL
> prefix.
>
> Happy to review the next revision, and thanks again to both of you for
> driving this.
>
> Regards,
> Prithvi S
>
> On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <dh...@apache.org> wrote:
>
> > As discussed live, looks like we both might have some drafts we can
> compare
> > and choose/merge from :)
> >
> > Here's my current draft spec PR; it opts for my preferred syntax choices
> > for now (/api/management/ for share-management, /api/shares/ for consumer
> > data-plane): https://github.com/apache/polaris/pull/5446
> >
> > Since we were talking about preferring mdfiles for design iteration
> here's
> > a companion design-addendum which just enumerates what the end-to-end
> user
> > journeys look like syntactically for each combination of design choices:
> >
> >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> >
> > Please let me know if anyone prefers a Google Doc version of the
> > design-addendum for easier commenting, or if the design-addendum itself
> > should also be modeled as a PR on some free-floating branch just to
> capture
> > comments.
> >
> >
> > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I am resuming work on this proposal.
> > >
> > > Regarding our recent discussion, I will move forward with opening a
> > > draft PR for the /api/shares/v1 endpoint to help illustrate the
> > > design.
> > >
> > > I will update this thread once the PR is ready.
> > >
> > > Thanks,
> > > JB
> > >
> > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > > >
> > > > Hi Dennis,
> > > >
> > > > Thanks for the recap!
> > > >
> > > > I was thinking that share management is probably distinct from
> catalog
> > > > management. A user who already has a realm+catalog may have an option
> > to
> > > > manage shares, but not the catalog itself. Would it make sense to
> > > separate
> > > > the share management API from the general Polaris management API? I
> > mean
> > > > using a different prefix for URIs.
> > > >
> > > > For example: /api/shares/v1/
> > > >
> > > > Obviously clients should not make any assumptions about the base path
> > > > (/api/shares), which could be different in different deployments
> (e.g.
> > > > going through a gateway). Clients must be able to take the base URI
> as
> > a
> > > > config option. Path segments after /v1/ will be specified by Polaris
> > > (Open
> > > > API yaml).
> > > >
> > > > WDYT?
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <dh...@apache.org> wrote:
> > > >
> > > > > Just to recap some of the discussion from today's community sync,
> > > Dmitri,
> > > > > Robert, and Alex helped raise a few considerations to roll into the
> > > next
> > > > > design/prototyping phase:
> > > > >
> > > > > 1. Given the way sharing is packaged as a separate thing, it may
> not
> > be
> > > > > obvious to users that historical snapshots are equally available to
> > > > > consumers as the latest snapshot (by virtue of sharing via the
> > > standard IRC
> > > > > protocol) - we should make sure to document that fact well,
> including
> > > the
> > > > > pitfall that deleting rows by updating a table doesn't necessarily
> > mean
> > > > > those rows become inaccessible to consumers
> > > > > 2. Make sure not to hard-code assumptions about the existence of an
> > > > > internal PrincipalEntity backing an ExternalConsumer - even if the
> > > first
> > > > > MVP implementation uses a nested/hidden Principal of some type that
> > > holds a
> > > > > Polaris-managed client_id/client_secret, the 3rd-party IdP use
> cases
> > > > > require "implicit" Principals resolved via the oauth token. In such
> > > cases
> > > > > the ExternalConsumer contains the metadata coordinating the
> three-way
> > > > > handshake between the data-provider, data-consumer, and IdP, but
> > there
> > > is
> > > > > no PrincipalEntity underlying it.
> > > > > 3. Longer-term, we may want to tackle "on-behalf-of" semantics -
> > > initially,
> > > > > the way an ExternalConsumer works is that the finer-grained
> consumer
> > > users
> > > > > share a single ExternalConsumer identity, just like how a FEDERATED
> > > catalog
> > > > > shares a single service principal in its ConnectionConfig. But in
> > some
> > > > > cases the consumer may want some end-to-end attribution of
> > fine-grained
> > > > > users, like "on-behalf-of: bob @ consumercompany.com" when the
> > > > > ExternalConsumer implicit principal is something like
> > > > > "externalconsumer$consumercompany"
> > > > > 4. We can iterate more rapidly on a draft spec PR and also expand
> on
> > > the
> > > > > detailed technical design
> > > > >
> > > > > Cheers,
> > > > > Dennis
> > > > >
> > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <dh...@apache.org>
> wrote:
> > > > >
> > > > > > Thanks JB for reviving this thread, and belated thanks to all
> > > reviewers
> > > > > > who have left feedback in the doc or this thread so far! And
> > > apologies
> > > > > for
> > > > > > the silence on this, it's still been on my radar but I got
> > > distracted by
> > > > > > some other work for awhile.
> > > > > >
> > > > > > Echoing JB's summary, the proposal is indeed that we're mostly
> just
> > > > > > reusing existing Authz/RBAC machinery, and the translation layer
> to
> > > the
> > > > > new
> > > > > > sharing concepts sits *on top of* the existing authz engine as
> much
> > > as
> > > > > > possible.
> > > > > >
> > > > > > So even if we're adding layers of indirection on
> > > ExternalConsumer/Listing
> > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the parts that end
> > up
> > > > > doing
> > > > > > the underlying authz just reuse existing Authorizer logic.
> > > > > >
> > > > > > Based on some other offline discussion with Dmitri, I realized
> why
> > > the
> > > > > > discussion around "requirements" may have been confusing:
> > > > > >
> > > > > > 1. My attempts to shorten the doc inadvertently cut out some of
> the
> > > more
> > > > > > "abstract requirements" formulations
> > > > > > 2. The project in itself is somewhat more of a "form *as*
> function"
> > > > > > feature than *strictly* functional in nature -- for example, many
> > > > > correctly
> > > > > > pointed out that "sharing" via the existing RBAC/Principals
> syntax
> > is
> > > > > > already *possible* just by letting a cross-org "consumer" be a
> > > > > first-class
> > > > > > principal and access your own company's main Polaris catalog
> > > endpoints.
> > > > > >
> > > > > > For (1) I added this to the Requirements section in the doc to
> > better
> > > > > > clarify the abstract form of the data model element:
> > > > > >
> > > > > > *A. Sharing Container*: Container concept for holding things you
> > > want to
> > > > > > share - “ShareEntity”
> > > > > > *B. Consumer Identity*: Special flavor of principal-like entity
> > > > > > representing a consumer relationship - “ExternalConsumer”
> > > > > > *C. Map Containers to Consumers*: A way to specify letting a
> given
> > > > > > consumer access a given sharing container - “Listing”
> > > > > > *D. Consumer Endpoint*: A segregated API access-point for
> > consumers -
> > > > > > “EndpointConfig”
> > > > > > *E. Objects in Containers*: A way to specify which in-catalog
> > objects
> > > > > > belong in the big sharing container - “ShareMembership”
> > > > > >
> > > > > > This still bakes in a bit of implementation detail since it begs
> > the
> > > > > > question why not just let core RBAC/grants handle (C) and (E). To
> > > that
> > > > > end,
> > > > > > and addressing problem (2) of where the "form *is* function", I
> > > added a
> > > > > > "Concept Mapping" tab:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > >
> > > > > > I think the details of more advanced features can still be hashed
> > out
> > > > > > incrementally without blocking work on an MVP (and certainly the
> > MVP
> > > > > scope
> > > > > > could be trimmed so we can launch-and-iterate), so as long as
> > there's
> > > > > > community alignment on the general direction I propose we could
> > move
> > > > > > forward on basic building blocks. Or if building blocks
> themselves
> > > are
> > > > > > controversial we could still proceed with some prototyping across
> > > > > different
> > > > > > options.
> > > > > >
> > > > > > Cheers,
> > > > > > Dennis
> > > > > >
> > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <
> > > jb@nanthrax.net>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi everyone,
> > > > > >>
> > > > > >> I would like to resume the discussion about Data Sharing in
> > Polaris
> > > > > >> and propose focusing on three major items to help us move
> forward.
> > > > > >>
> > > > > >> 1. What is data sharing?
> > > > > >> In a nutshell, it is an open protocol proposal allowing
> > > > > >> organizations to share live data without copying or replication.
> > The
> > > > > >> main goal is zero-copy sharing achieved through shared metadata
> > and
> > > > > >> authorization.
> > > > > >>
> > > > > >> On Polaris, this would mean:
> > > > > >> a. The consumer is authenticated as usual.
> > > > > >> b. The Polaris server provides the consumer with direct access
> to
> > > the
> > > > > >> data files (e.g., Parquet, as specified in the metadata.json).
> > > > > >> c. The consumer reads the data files directly using their
> > preferred
> > > > > >> tool (e.g., a query engine).
> > > > > >>
> > > > > >> While Polaris already has most of the necessary underlying
> > > > > >> capabilities, we need clear branding/packaging around this use
> > case.
> > > > > >> Specifically, this means:
> > > > > >> a. A share is a dedicated catalog in the Polaris server, or a
> > > catalog
> > > > > >> role scoped to specific namespaces.
> > > > > >> b. A recipient is represented by a principal and a principal
> role.
> > > > > >> c. A consumer profile consists of the catalog URI and the OAuth2
> > > > > >> client_id/client_secret.
> > > > > >> d. We use credential vending (temporary STS credentials) for
> > > consumer
> > > > > >> access to the data files.
> > > > > >> e. We use the Polaris IRC layer to expose the metadata and the
> > file
> > > > > >> locations/access.
> > > > > >> f. We can also support Polaris-to-Polaris sharing via
> Federation.
> > > > > >>
> > > > > >> Concretely:
> > > > > >>
> > > > > >> - On the provider side, we use the standard Polaris RBAC model
> > > > > >> (assigning grants to catalog roles, and assigning those catalog
> > > roles
> > > > > >> to principal roles):
> > > > > >> - Create a dedicated principal with its own
> > > > > client_id/client_secret.
> > > > > >> - Create a catalog role with TABLE_READ_DATA, TABLE_LIST,
> > and
> > > > > >> NAMESPACE_LIST grants on the namespaces to be shared (no write
> > > > > >> grants).
> > > > > >> - Create a principal role per destination, attach the
> > catalog
> > > > > >> role, and assign the principal.
> > > > > >> - The destination engine uses loadTable with the
> > > > > >> "X-Iceberg-Access-Delegation: vended-credentials" header.
> Polaris
> > > then
> > > > > >> emits temporary, read-only storage credentials, allowing the
> > engine
> > > to
> > > > > >> read the data files directly from the owner's bucket without
> > > accessing
> > > > > >> the rest of the bucket. This achieves zero-copy sharing.
> > > > > >> - On the recipient/consumer side, no special configuration is
> > > > > >> needed; any IRC client can use it.
> > > > > >>
> > > > > >> 2. What is data sharing not?
> > > > > >> I previously considered adding an Arrow Flight endpoint to
> > > > > >> Polaris, but I no longer think it is a good idea for the initial
> > > > > >> phase. An Arrow Flight endpoint would serve the data directly,
> > which
> > > > > >> would require Polaris to read the data and expose it. I am
> against
> > > > > >> embedding a query engine in Polaris, as this tight coupling
> feels
> > > like
> > > > > >> an anti-pattern. While we could use a delegation service to
> serve
> > > the
> > > > > >> Flight endpoint, I think it should be considered for a later
> > phase,
> > > if
> > > > > >> at all, since a Flight endpoint might be better suited as a
> > consumer
> > > > > >> responsibility and could encourage data copying.
> > > > > >>
> > > > > >> The first phase should focus on packaging and branding the
> > > > > >> capabilities we already have.
> > > > > >>
> > > > > >> 3. What could the Data Sharing "package" look like in Polaris?
> > > > > >> To make Data Sharing obvious and easy to use, we should
> > > introduce
> > > > > >> the missing concept of a "share"—a named definition that users
> can
> > > > > >> list, discover, and define. The goal is to materialize the
> user's
> > > > > >> intent ("I want to share these tables with this consumer until
> > this
> > > > > >> date") into standard Polaris entities (principal, principal
> role,
> > > > > >> catalog role, and grants).
> > > > > >>
> > > > > >> Therefore, I propose adding a /shares endpoint to the Polaris
> > > > > >> Management API to package and materialize these share
> definitions
> > > into
> > > > > >> Polaris entities. This would provide a usable, first-version
> > package
> > > > > >> for Polaris Data Sharing.
> > > > > >>
> > > > > >> I hope this addresses Dmitri's questions and provides a more
> > > concrete
> > > > > >> scenario for Polaris Data Sharing.
> > > > > >>
> > > > > >> Thoughts?
> > > > > >>
> > > > > >> Regards,
> > > > > >> JB
> > > > > >>
> > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <dh...@apache.org>
> > wrote:
> > > > > >> >
> > > > > >> > Hi All,
> > > > > >> >
> > > > > >> > One big emerging enterprise use case coming up as more people
> > > > > >> consolidate
> > > > > >> > Data Lakehouses and Catalogs is something commonly known as
> > "Data
> > > > > >> Sharing",
> > > > > >> > an more specifically over the course of adoption of open table
> > > formats
> > > > > >> > "Open Data Sharing".
> > > > > >> >
> > > > > >> > Examples of existing managed service providers' Data Sharing
> > > features:
> > > > > >> >
> > > > > >> > https://www.databricks.com/product/delta-sharing
> > > > > >> > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > >> >
> > > > >
> > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > >> >
> > > > >
> > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > >> >
> > > > > >> > The basic idea is that when you share data between different
> > > > > companies,
> > > > > >> you
> > > > > >> > need a first-class governance/management layer and extra
> > > > > >> bells-and-whistles
> > > > > >> > that are distinct from just the basic capabilities of RBAC or
> > > > > >> generalized
> > > > > >> > access-control (i.e. if you're sharing across
> > partially-untrusted
> > > org
> > > > > >> > boundaries, you don't just let the consumer organization log
> > into
> > > your
> > > > > >> > datalake like one of your own employees).
> > > > > >> >
> > > > > >> > JB and I put together this high-level proposal for supporting
> > Open
> > > > > >> Sharing
> > > > > >> > in Polaris:
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > >> >
> > > > > >> > Tentatively, it means adding ~5 logical data model constructs,
> > > some of
> > > > > >> > which may be a first-class PolarisEntity type, others subtypes
> > of
> > > > > >> existing
> > > > > >> > entities, and others just a nested construct:
> > > > > >> >
> > > > > >> > - ShareEntity (would behave similarly to a Catalog)
> > > > > >> > - ExternalConsumer (mostly inherits from Principal)
> > > > > >> > - Listing (Similar to a "role grant" but has different
> > > metadata)
> > > > > >> > - EndpointConfig (nested config under Listing)
> > > > > >> > - ShareMembership (Similar to a "securable grant" but
> > different
> > > > > >> metadata)
> > > > > >> >
> > > > > >> > Feedback/comments welcome! I'll also bring it up for live
> > > discussion
> > > > > if
> > > > > >> > there's time in the community sync.
> > > > > >> >
> > > > > >> > Cheers,
> > > > > >> > Dennis
> > > > > >>
> > > > > >
> > > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dennis Huo <dh...@apache.org>.
Thanks for the review Prithvi! All great questions. Also, I realize I
forgot to address Dmitri's comment:
Would it make sense to separate
> the share management API from the general Polaris management API? I mean
> using a different prefix for URIs.
>
> For example: /api/shares/v1/
Right now IIRC the mental model is that "everything considered data plane"
ends up under /api/catalog/ and "everything considered admin plane" ends up
under /api/management/, and basically spans all the RBAC *and* catalogs.
I agree the *privileges* themselves should be separated between catalog
stuff, principal/role stuff, and sharing stuff.
Some pointers for posterity since it's a bit hard to keep track of how the
basePath that causes /api/catalog/ is kind of hidden in
https://github.com/apache/polaris/blob/7866ce5c009d64e4c1454b47bca3df9873c6c0b2/api/iceberg-service/build.gradle.kts#L86
:
serverVariables.put("basePath", "api/catalog")
while the /api/management is hard-coded into the yaml:
https://github.com/apache/polaris/blob/b42f931a461f6b96512f7de6955c7c09b5e413fa/spec/polaris-management-service.yml#L26
servers:
- url: "{scheme}://{host}/api/management/v1"
Using "/api/shares/" adds an orthogonal carveout. At the very least I think
we'd want to keep share-management segregated from share-data-plane.
For the data plane, remapping that under "/api/shares/v1" makes sense since
we want the entire subtree of share-related *data plane* operations to
match the layout of a catalog, so we're basically saying a consumer client
can just get a catalogBasePath of /api/shares/v1/ instead of
/api/catalog/v1/ and everything under it is the same.
If we did split management out, I guess we'd put it under
"/api/shares-management/v1/" instead or similar. This is plausible, though
the argument in favor of keeping it in /api/management/v1/ is that therre
are referential relatinships between the shares and catalogs, and along
with the way Listings/ExternalConsumers behave like other management-layer
counterparts, these all belong to the same "family" of API surfaces.
The yaml files themselves can still be separate to help
manageability/modularity of the codebase, the same way we have various
/api/catalog/v1/ extensions split into different yamls.
I don't feel too strongly though as long as we segregate the prefix for
control-plane vs data-plane. WDYT? Do you prefer /api/shares-management/v1/
instead of addint to /api/management/v1?
Other responses to Prithvi's questions inline:
1. Membership as source of truth. If a hidden catalog role is independently
> grantable, a share could be widened without going through the share APIs.
> Would it make sense to say the data plane authorizes against membership +
> listing, and that any backing roles are not user-visible / not
> independently grantable?
>
Yes, good point; if we use the grants as a hidden implementation detail we
need to make sure they're only modified by the share-management system.
The precedent for this is the "system$" reserved-prefix that was briefly
discussed most recently here
https://github.com/apache/polaris/pull/4629#discussion_r3372018934
I think we don't yet have ingrained usage of it beyond customized
provider-specific versions of managed Polaris, so we'll need to be careful
that the "system protection" around these entities holds up, but it's at
least possible to do.
> 2. ExternalConsumer and principals. The Aug 7 recap already noted we should
> not assume a PrincipalEntity behind every ExternalConsumer. The journeys
> still describe granting the consumer a share principal role, I assume that
> is an implementation sketch for v1 client_id/secret, not the public model?
>
Good nuance, I admittedly gloss over representation of federated identities
as ExternalConsumers in the current journeys doc, and I need to make sure
the federated-identity case is called out as one of the caveats for the
PrincipalRole approach (compared to the options where a share-entity is
literally defined as a *child* of a catatalog, and only a CatalogRole needs
to participate).
Note that the client_id/secret approach is still part of the "public"
model, but I'l assume you specifically mean the *federated* public model.
For the federated case, it should still be possible to use the system$ type
of hidden PrincipalRole, we just need the resolution of the share-attached
PrincipalRole *not* to rely on the GrantPrinciplRoleToPrincipal (where the
principal would be an ExternalConsumer's hidden principal in the v1 case),
but allow the *Listing* itself to be the surrogate resolution path from a
virtual-Principal (the federated identity) to the same hidden share
PrincipalRole.
This is a good design detail to track though - whether the
ExternalConsumer<->(Hidden) PrincipalRole is statically resolved at
Listing-creation time or by dynamically evaluating the Listing at
consumer-query time. We'd need a way to resolve the listing from the
federated auth flow.
> 3. Listing expiration. JB's original note was "share these tables with this
> consumer until this date." Credential expiry is a bit different. Would a
> listing-level expiration (and maybe ACTIVE/SUSPENDED) be in scope for v1?
>
Generally I'd say v1 scope should be just focused on the minimal set of
functionality needed, so by default I definitely wouldn't say it's a
"blocker" for v1, but I guess we can still debate relative prioritization
against other bells-and-whistles.
If the nature of the sharing use cases makes this a high-demand feature
there's no reason we can't try to add it to earlier scope.
> 4. Time-travel. As raised in the community sync, sharing via Iceberg REST
> means historical snapshots remain readable. Totally fine for v1 to support
> only full history, would an explicit field on the share help so a later
> CURRENT_ONLY option is not a breaking surprise?
>
Yeah, I think the most important thing is to have a line-of-sight up front
for how we *could* implement narrower privileges.
Offhand it seems like in this case adding a field later should be easy to
do in a non-breaking way, but I don't feel too strongly. We could make it
an enum with only one available option for now, ALL_HISTORICAL_SNAPSHOTS.
Prerequisites would probably be full ScanPlan API support and HTTP
presigned URL support in Iceberg so that we could vend file-granularity
credentials.
> 5. Consumer data plane. The current file is control-plane only, which is a
> good first step. It might help to list the Iceberg routes that are mounted,
> that writes 404, that non-members 404, and that /config does not return
> provider catalog properties, so implementations don't diverge.
>
Good catch, I was actually just working on some prototyping for whether to
autogen vs copy/paste parts of the data-plane api definitions. I'll post a
followup shortly.
Looks like there's also a related fix even for the basic
non-sharing-related GetCatalog case where we need to tighten the semantics
of whether /config returnns catalog properties:
https://github.com/apache/polaris/pull/5246
This will be a good category of design pitfalls to keep track of where we
need to be careful to limit intra-entity segregation of intended
access/exposure through a share (for example, whether table-default.*
properties set on a Catalog or Namespace should ever be served up in a
LoadTableResponse when accessed through a Share)
>
> Two smaller notes, if useful: in the J1 journey the consumer still posts to
> /api/catalog/v1/oauth/tokens, if partners are not supposed to touch the
> internal catalog API, should the default token endpoint live on the shares
> surface (or an external IdP)? And on Dmitri's point about share operators
> vs catalog operators, a SHARE_MANAGE privilege on the bound catalog
> (independent of CATALOG_MANAGE_ACCESS) would make that work on any URL
> prefix.
>
>
Long term probably want everything to converge more on external IdP. But
you're right that as long as we *do* support the Polaris-independent OAuth
cases (which I think we will for the foreseeable future), mounting under
/api/shares/v1/oauth/tokens is probably best.
In concept the in-catalog OAuth deprecation in the Iceberg REST spec
doesn't necessarily preclude provider self-owned OAuth, just that the
Iceberg layer doesn't mandate it; so the "clean" formulation would be that
the catalog provider who *chooses* to also be the oauth provider (in this
case, a self-sufficient Polaris instance) could choose to
put their token URL at a different basePath entirely like
/api/iam/v1/oauth/tokens, and one could argue whether catalog-access and
shares-access should share that one.
But for now while we still support the version of the IRC spec that *does*
have the oauth endpoint defined, I guess the best of all worlds is indeed
to have /api/catalog/v1/oauth/tokens be segregated from
/api/shares/v1/oauth/tokens, with the coincidental benefit of better
structural segregation for the sharing consumers anyways.
Good point about the privileges, I agree that having the right
target/secondary privileges for share-related AuthorizableOperations should
enable segregating the personas.
> Happy to review the next revision, and thanks again to both of you for
> driving this.
>
> Regards,
> Prithvi S
>
> On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <dh...@apache.org> wrote:
>
> > As discussed live, looks like we both might have some drafts we can
> compare
> > and choose/merge from :)
> >
> > Here's my current draft spec PR; it opts for my preferred syntax choices
> > for now (/api/management/ for share-management, /api/shares/ for consumer
> > data-plane): https://github.com/apache/polaris/pull/5446
> >
> > Since we were talking about preferring mdfiles for design iteration
> here's
> > a companion design-addendum which just enumerates what the end-to-end
> user
> > journeys look like syntactically for each combination of design choices:
> >
> >
> >
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
> >
> > Please let me know if anyone prefers a Google Doc version of the
> > design-addendum for easier commenting, or if the design-addendum itself
> > should also be modeled as a PR on some free-floating branch just to
> capture
> > comments.
> >
> >
> > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I am resuming work on this proposal.
> > >
> > > Regarding our recent discussion, I will move forward with opening a
> > > draft PR for the /api/shares/v1 endpoint to help illustrate the
> > > design.
> > >
> > > I will update this thread once the PR is ready.
> > >
> > > Thanks,
> > > JB
> > >
> > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > > >
> > > > Hi Dennis,
> > > >
> > > > Thanks for the recap!
> > > >
> > > > I was thinking that share management is probably distinct from
> catalog
> > > > management. A user who already has a realm+catalog may have an option
> > to
> > > > manage shares, but not the catalog itself. Would it make sense to
> > > separate
> > > > the share management API from the general Polaris management API? I
> > mean
> > > > using a different prefix for URIs.
> > > >
> > > > For example: /api/shares/v1/
> > > >
> > > > Obviously clients should not make any assumptions about the base path
> > > > (/api/shares), which could be different in different deployments
> (e.g.
> > > > going through a gateway). Clients must be able to take the base URI
> as
> > a
> > > > config option. Path segments after /v1/ will be specified by Polaris
> > > (Open
> > > > API yaml).
> > > >
> > > > WDYT?
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <dh...@apache.org> wrote:
> > > >
> > > > > Just to recap some of the discussion from today's community sync,
> > > Dmitri,
> > > > > Robert, and Alex helped raise a few considerations to roll into the
> > > next
> > > > > design/prototyping phase:
> > > > >
> > > > > 1. Given the way sharing is packaged as a separate thing, it may
> not
> > be
> > > > > obvious to users that historical snapshots are equally available to
> > > > > consumers as the latest snapshot (by virtue of sharing via the
> > > standard IRC
> > > > > protocol) - we should make sure to document that fact well,
> including
> > > the
> > > > > pitfall that deleting rows by updating a table doesn't necessarily
> > mean
> > > > > those rows become inaccessible to consumers
> > > > > 2. Make sure not to hard-code assumptions about the existence of an
> > > > > internal PrincipalEntity backing an ExternalConsumer - even if the
> > > first
> > > > > MVP implementation uses a nested/hidden Principal of some type that
> > > holds a
> > > > > Polaris-managed client_id/client_secret, the 3rd-party IdP use
> cases
> > > > > require "implicit" Principals resolved via the oauth token. In such
> > > cases
> > > > > the ExternalConsumer contains the metadata coordinating the
> three-way
> > > > > handshake between the data-provider, data-consumer, and IdP, but
> > there
> > > is
> > > > > no PrincipalEntity underlying it.
> > > > > 3. Longer-term, we may want to tackle "on-behalf-of" semantics -
> > > initially,
> > > > > the way an ExternalConsumer works is that the finer-grained
> consumer
> > > users
> > > > > share a single ExternalConsumer identity, just like how a FEDERATED
> > > catalog
> > > > > shares a single service principal in its ConnectionConfig. But in
> > some
> > > > > cases the consumer may want some end-to-end attribution of
> > fine-grained
> > > > > users, like "on-behalf-of: bob @ consumercompany.com" when the
> > > > > ExternalConsumer implicit principal is something like
> > > > > "externalconsumer$consumercompany"
> > > > > 4. We can iterate more rapidly on a draft spec PR and also expand
> on
> > > the
> > > > > detailed technical design
> > > > >
> > > > > Cheers,
> > > > > Dennis
> > > > >
> > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <dh...@apache.org>
> wrote:
> > > > >
> > > > > > Thanks JB for reviving this thread, and belated thanks to all
> > > reviewers
> > > > > > who have left feedback in the doc or this thread so far! And
> > > apologies
> > > > > for
> > > > > > the silence on this, it's still been on my radar but I got
> > > distracted by
> > > > > > some other work for awhile.
> > > > > >
> > > > > > Echoing JB's summary, the proposal is indeed that we're mostly
> just
> > > > > > reusing existing Authz/RBAC machinery, and the translation layer
> to
> > > the
> > > > > new
> > > > > > sharing concepts sits *on top of* the existing authz engine as
> much
> > > as
> > > > > > possible.
> > > > > >
> > > > > > So even if we're adding layers of indirection on
> > > ExternalConsumer/Listing
> > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the parts that end
> > up
> > > > > doing
> > > > > > the underlying authz just reuse existing Authorizer logic.
> > > > > >
> > > > > > Based on some other offline discussion with Dmitri, I realized
> why
> > > the
> > > > > > discussion around "requirements" may have been confusing:
> > > > > >
> > > > > > 1. My attempts to shorten the doc inadvertently cut out some of
> the
> > > more
> > > > > > "abstract requirements" formulations
> > > > > > 2. The project in itself is somewhat more of a "form *as*
> function"
> > > > > > feature than *strictly* functional in nature -- for example, many
> > > > > correctly
> > > > > > pointed out that "sharing" via the existing RBAC/Principals
> syntax
> > is
> > > > > > already *possible* just by letting a cross-org "consumer" be a
> > > > > first-class
> > > > > > principal and access your own company's main Polaris catalog
> > > endpoints.
> > > > > >
> > > > > > For (1) I added this to the Requirements section in the doc to
> > better
> > > > > > clarify the abstract form of the data model element:
> > > > > >
> > > > > > *A. Sharing Container*: Container concept for holding things you
> > > want to
> > > > > > share - “ShareEntity”
> > > > > > *B. Consumer Identity*: Special flavor of principal-like entity
> > > > > > representing a consumer relationship - “ExternalConsumer”
> > > > > > *C. Map Containers to Consumers*: A way to specify letting a
> given
> > > > > > consumer access a given sharing container - “Listing”
> > > > > > *D. Consumer Endpoint*: A segregated API access-point for
> > consumers -
> > > > > > “EndpointConfig”
> > > > > > *E. Objects in Containers*: A way to specify which in-catalog
> > objects
> > > > > > belong in the big sharing container - “ShareMembership”
> > > > > >
> > > > > > This still bakes in a bit of implementation detail since it begs
> > the
> > > > > > question why not just let core RBAC/grants handle (C) and (E). To
> > > that
> > > > > end,
> > > > > > and addressing problem (2) of where the "form *is* function", I
> > > added a
> > > > > > "Concept Mapping" tab:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > > >
> > > > > > I think the details of more advanced features can still be hashed
> > out
> > > > > > incrementally without blocking work on an MVP (and certainly the
> > MVP
> > > > > scope
> > > > > > could be trimmed so we can launch-and-iterate), so as long as
> > there's
> > > > > > community alignment on the general direction I propose we could
> > move
> > > > > > forward on basic building blocks. Or if building blocks
> themselves
> > > are
> > > > > > controversial we could still proceed with some prototyping across
> > > > > different
> > > > > > options.
> > > > > >
> > > > > > Cheers,
> > > > > > Dennis
> > > > > >
> > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <
> > > jb@nanthrax.net>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi everyone,
> > > > > >>
> > > > > >> I would like to resume the discussion about Data Sharing in
> > Polaris
> > > > > >> and propose focusing on three major items to help us move
> forward.
> > > > > >>
> > > > > >> 1. What is data sharing?
> > > > > >> In a nutshell, it is an open protocol proposal allowing
> > > > > >> organizations to share live data without copying or replication.
> > The
> > > > > >> main goal is zero-copy sharing achieved through shared metadata
> > and
> > > > > >> authorization.
> > > > > >>
> > > > > >> On Polaris, this would mean:
> > > > > >> a. The consumer is authenticated as usual.
> > > > > >> b. The Polaris server provides the consumer with direct access
> to
> > > the
> > > > > >> data files (e.g., Parquet, as specified in the metadata.json).
> > > > > >> c. The consumer reads the data files directly using their
> > preferred
> > > > > >> tool (e.g., a query engine).
> > > > > >>
> > > > > >> While Polaris already has most of the necessary underlying
> > > > > >> capabilities, we need clear branding/packaging around this use
> > case.
> > > > > >> Specifically, this means:
> > > > > >> a. A share is a dedicated catalog in the Polaris server, or a
> > > catalog
> > > > > >> role scoped to specific namespaces.
> > > > > >> b. A recipient is represented by a principal and a principal
> role.
> > > > > >> c. A consumer profile consists of the catalog URI and the OAuth2
> > > > > >> client_id/client_secret.
> > > > > >> d. We use credential vending (temporary STS credentials) for
> > > consumer
> > > > > >> access to the data files.
> > > > > >> e. We use the Polaris IRC layer to expose the metadata and the
> > file
> > > > > >> locations/access.
> > > > > >> f. We can also support Polaris-to-Polaris sharing via
> Federation.
> > > > > >>
> > > > > >> Concretely:
> > > > > >>
> > > > > >> - On the provider side, we use the standard Polaris RBAC model
> > > > > >> (assigning grants to catalog roles, and assigning those catalog
> > > roles
> > > > > >> to principal roles):
> > > > > >> - Create a dedicated principal with its own
> > > > > client_id/client_secret.
> > > > > >> - Create a catalog role with TABLE_READ_DATA, TABLE_LIST,
> > and
> > > > > >> NAMESPACE_LIST grants on the namespaces to be shared (no write
> > > > > >> grants).
> > > > > >> - Create a principal role per destination, attach the
> > catalog
> > > > > >> role, and assign the principal.
> > > > > >> - The destination engine uses loadTable with the
> > > > > >> "X-Iceberg-Access-Delegation: vended-credentials" header.
> Polaris
> > > then
> > > > > >> emits temporary, read-only storage credentials, allowing the
> > engine
> > > to
> > > > > >> read the data files directly from the owner's bucket without
> > > accessing
> > > > > >> the rest of the bucket. This achieves zero-copy sharing.
> > > > > >> - On the recipient/consumer side, no special configuration is
> > > > > >> needed; any IRC client can use it.
> > > > > >>
> > > > > >> 2. What is data sharing not?
> > > > > >> I previously considered adding an Arrow Flight endpoint to
> > > > > >> Polaris, but I no longer think it is a good idea for the initial
> > > > > >> phase. An Arrow Flight endpoint would serve the data directly,
> > which
> > > > > >> would require Polaris to read the data and expose it. I am
> against
> > > > > >> embedding a query engine in Polaris, as this tight coupling
> feels
> > > like
> > > > > >> an anti-pattern. While we could use a delegation service to
> serve
> > > the
> > > > > >> Flight endpoint, I think it should be considered for a later
> > phase,
> > > if
> > > > > >> at all, since a Flight endpoint might be better suited as a
> > consumer
> > > > > >> responsibility and could encourage data copying.
> > > > > >>
> > > > > >> The first phase should focus on packaging and branding the
> > > > > >> capabilities we already have.
> > > > > >>
> > > > > >> 3. What could the Data Sharing "package" look like in Polaris?
> > > > > >> To make Data Sharing obvious and easy to use, we should
> > > introduce
> > > > > >> the missing concept of a "share"—a named definition that users
> can
> > > > > >> list, discover, and define. The goal is to materialize the
> user's
> > > > > >> intent ("I want to share these tables with this consumer until
> > this
> > > > > >> date") into standard Polaris entities (principal, principal
> role,
> > > > > >> catalog role, and grants).
> > > > > >>
> > > > > >> Therefore, I propose adding a /shares endpoint to the Polaris
> > > > > >> Management API to package and materialize these share
> definitions
> > > into
> > > > > >> Polaris entities. This would provide a usable, first-version
> > package
> > > > > >> for Polaris Data Sharing.
> > > > > >>
> > > > > >> I hope this addresses Dmitri's questions and provides a more
> > > concrete
> > > > > >> scenario for Polaris Data Sharing.
> > > > > >>
> > > > > >> Thoughts?
> > > > > >>
> > > > > >> Regards,
> > > > > >> JB
> > > > > >>
> > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <dh...@apache.org>
> > wrote:
> > > > > >> >
> > > > > >> > Hi All,
> > > > > >> >
> > > > > >> > One big emerging enterprise use case coming up as more people
> > > > > >> consolidate
> > > > > >> > Data Lakehouses and Catalogs is something commonly known as
> > "Data
> > > > > >> Sharing",
> > > > > >> > an more specifically over the course of adoption of open table
> > > formats
> > > > > >> > "Open Data Sharing".
> > > > > >> >
> > > > > >> > Examples of existing managed service providers' Data Sharing
> > > features:
> > > > > >> >
> > > > > >> > https://www.databricks.com/product/delta-sharing
> > > > > >> > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > > >> >
> > > > >
> > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > > >> >
> > > > >
> > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > > >> >
> > > > > >> > The basic idea is that when you share data between different
> > > > > companies,
> > > > > >> you
> > > > > >> > need a first-class governance/management layer and extra
> > > > > >> bells-and-whistles
> > > > > >> > that are distinct from just the basic capabilities of RBAC or
> > > > > >> generalized
> > > > > >> > access-control (i.e. if you're sharing across
> > partially-untrusted
> > > org
> > > > > >> > boundaries, you don't just let the consumer organization log
> > into
> > > your
> > > > > >> > datalake like one of your own employees).
> > > > > >> >
> > > > > >> > JB and I put together this high-level proposal for supporting
> > Open
> > > > > >> Sharing
> > > > > >> > in Polaris:
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > > >> >
> > > > > >> > Tentatively, it means adding ~5 logical data model constructs,
> > > some of
> > > > > >> > which may be a first-class PolarisEntity type, others subtypes
> > of
> > > > > >> existing
> > > > > >> > entities, and others just a nested construct:
> > > > > >> >
> > > > > >> > - ShareEntity (would behave similarly to a Catalog)
> > > > > >> > - ExternalConsumer (mostly inherits from Principal)
> > > > > >> > - Listing (Similar to a "role grant" but has different
> > > metadata)
> > > > > >> > - EndpointConfig (nested config under Listing)
> > > > > >> > - ShareMembership (Similar to a "securable grant" but
> > different
> > > > > >> metadata)
> > > > > >> >
> > > > > >> > Feedback/comments welcome! I'll also bring it up for live
> > > discussion
> > > > > if
> > > > > >> > there's time in the community sync.
> > > > > >> >
> > > > > >> > Cheers,
> > > > > >> > Dennis
> > > > > >>
> > > > > >
> > > > >
> > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Prithvi S <pr...@gmail.com>.
Hi everyone,
Thank you for reviving this, and for the concrete spec in PR 5446.. the
journeys doc was especially helpful for seeing the end-to-end flow.
I agree this is worth doing, and that v1 is more than a tutorial on today's
RBAC. A partner principal on /api/catalog is already possible; what seems
valuable is a first-class share (enumerated members, a restricted consumer,
a listing as the binding and audit unit) plus a segregated, read-only
Iceberg REST surface with credential vending. Stock Iceberg REST as the
consumer protocol, and keeping Flight and Polaris-to-Polaris federation out
of v1, all sound right to me.
I don't have a strong view yet on the URL layout (management vs shares
prefix, nested vs top-level, prefix keyed by share vs listing) and am happy
to follow whatever the community prefers there.
A few things I wanted to check on the spec itself, several of these echo
points already raised on the list:
1. Membership as source of truth. If a hidden catalog role is independently
grantable, a share could be widened without going through the share APIs.
Would it make sense to say the data plane authorizes against membership +
listing, and that any backing roles are not user-visible / not
independently grantable?
2. ExternalConsumer and principals. The Aug 7 recap already noted we should
not assume a PrincipalEntity behind every ExternalConsumer. The journeys
still describe granting the consumer a share principal role, I assume that
is an implementation sketch for v1 client_id/secret, not the public model?
3. Listing expiration. JB's original note was "share these tables with this
consumer until this date." Credential expiry is a bit different. Would a
listing-level expiration (and maybe ACTIVE/SUSPENDED) be in scope for v1?
4. Time-travel. As raised in the community sync, sharing via Iceberg REST
means historical snapshots remain readable. Totally fine for v1 to support
only full history, would an explicit field on the share help so a later
CURRENT_ONLY option is not a breaking surprise?
5. Consumer data plane. The current file is control-plane only, which is a
good first step. It might help to list the Iceberg routes that are mounted,
that writes 404, that non-members 404, and that /config does not return
provider catalog properties, so implementations don't diverge.
Two smaller notes, if useful: in the J1 journey the consumer still posts to
/api/catalog/v1/oauth/tokens, if partners are not supposed to touch the
internal catalog API, should the default token endpoint live on the shares
surface (or an external IdP)? And on Dmitri's point about share operators
vs catalog operators, a SHARE_MANAGE privilege on the bound catalog
(independent of CATALOG_MANAGE_ACCESS) would make that work on any URL
prefix.
Happy to review the next revision, and thanks again to both of you for
driving this.
Regards,
Prithvi S
On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <dh...@apache.org> wrote:
> As discussed live, looks like we both might have some drafts we can compare
> and choose/merge from :)
>
> Here's my current draft spec PR; it opts for my preferred syntax choices
> for now (/api/management/ for share-management, /api/shares/ for consumer
> data-plane): https://github.com/apache/polaris/pull/5446
>
> Since we were talking about preferring mdfiles for design iteration here's
> a companion design-addendum which just enumerates what the end-to-end user
> journeys look like syntactically for each combination of design choices:
>
>
> https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
>
> Please let me know if anyone prefers a Google Doc version of the
> design-addendum for easier commenting, or if the design-addendum itself
> should also be modeled as a PR on some free-floating branch just to capture
> comments.
>
>
> On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi everyone,
> >
> > I am resuming work on this proposal.
> >
> > Regarding our recent discussion, I will move forward with opening a
> > draft PR for the /api/shares/v1 endpoint to help illustrate the
> > design.
> >
> > I will update this thread once the PR is ready.
> >
> > Thanks,
> > JB
> >
> > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> > >
> > > Hi Dennis,
> > >
> > > Thanks for the recap!
> > >
> > > I was thinking that share management is probably distinct from catalog
> > > management. A user who already has a realm+catalog may have an option
> to
> > > manage shares, but not the catalog itself. Would it make sense to
> > separate
> > > the share management API from the general Polaris management API? I
> mean
> > > using a different prefix for URIs.
> > >
> > > For example: /api/shares/v1/
> > >
> > > Obviously clients should not make any assumptions about the base path
> > > (/api/shares), which could be different in different deployments (e.g.
> > > going through a gateway). Clients must be able to take the base URI as
> a
> > > config option. Path segments after /v1/ will be specified by Polaris
> > (Open
> > > API yaml).
> > >
> > > WDYT?
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <dh...@apache.org> wrote:
> > >
> > > > Just to recap some of the discussion from today's community sync,
> > Dmitri,
> > > > Robert, and Alex helped raise a few considerations to roll into the
> > next
> > > > design/prototyping phase:
> > > >
> > > > 1. Given the way sharing is packaged as a separate thing, it may not
> be
> > > > obvious to users that historical snapshots are equally available to
> > > > consumers as the latest snapshot (by virtue of sharing via the
> > standard IRC
> > > > protocol) - we should make sure to document that fact well, including
> > the
> > > > pitfall that deleting rows by updating a table doesn't necessarily
> mean
> > > > those rows become inaccessible to consumers
> > > > 2. Make sure not to hard-code assumptions about the existence of an
> > > > internal PrincipalEntity backing an ExternalConsumer - even if the
> > first
> > > > MVP implementation uses a nested/hidden Principal of some type that
> > holds a
> > > > Polaris-managed client_id/client_secret, the 3rd-party IdP use cases
> > > > require "implicit" Principals resolved via the oauth token. In such
> > cases
> > > > the ExternalConsumer contains the metadata coordinating the three-way
> > > > handshake between the data-provider, data-consumer, and IdP, but
> there
> > is
> > > > no PrincipalEntity underlying it.
> > > > 3. Longer-term, we may want to tackle "on-behalf-of" semantics -
> > initially,
> > > > the way an ExternalConsumer works is that the finer-grained consumer
> > users
> > > > share a single ExternalConsumer identity, just like how a FEDERATED
> > catalog
> > > > shares a single service principal in its ConnectionConfig. But in
> some
> > > > cases the consumer may want some end-to-end attribution of
> fine-grained
> > > > users, like "on-behalf-of: bob @ consumercompany.com" when the
> > > > ExternalConsumer implicit principal is something like
> > > > "externalconsumer$consumercompany"
> > > > 4. We can iterate more rapidly on a draft spec PR and also expand on
> > the
> > > > detailed technical design
> > > >
> > > > Cheers,
> > > > Dennis
> > > >
> > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <dh...@apache.org> wrote:
> > > >
> > > > > Thanks JB for reviving this thread, and belated thanks to all
> > reviewers
> > > > > who have left feedback in the doc or this thread so far! And
> > apologies
> > > > for
> > > > > the silence on this, it's still been on my radar but I got
> > distracted by
> > > > > some other work for awhile.
> > > > >
> > > > > Echoing JB's summary, the proposal is indeed that we're mostly just
> > > > > reusing existing Authz/RBAC machinery, and the translation layer to
> > the
> > > > new
> > > > > sharing concepts sits *on top of* the existing authz engine as much
> > as
> > > > > possible.
> > > > >
> > > > > So even if we're adding layers of indirection on
> > ExternalConsumer/Listing
> > > > > -> Principal/PrincipalRole/CatalogRole+grants, the parts that end
> up
> > > > doing
> > > > > the underlying authz just reuse existing Authorizer logic.
> > > > >
> > > > > Based on some other offline discussion with Dmitri, I realized why
> > the
> > > > > discussion around "requirements" may have been confusing:
> > > > >
> > > > > 1. My attempts to shorten the doc inadvertently cut out some of the
> > more
> > > > > "abstract requirements" formulations
> > > > > 2. The project in itself is somewhat more of a "form *as* function"
> > > > > feature than *strictly* functional in nature -- for example, many
> > > > correctly
> > > > > pointed out that "sharing" via the existing RBAC/Principals syntax
> is
> > > > > already *possible* just by letting a cross-org "consumer" be a
> > > > first-class
> > > > > principal and access your own company's main Polaris catalog
> > endpoints.
> > > > >
> > > > > For (1) I added this to the Requirements section in the doc to
> better
> > > > > clarify the abstract form of the data model element:
> > > > >
> > > > > *A. Sharing Container*: Container concept for holding things you
> > want to
> > > > > share - “ShareEntity”
> > > > > *B. Consumer Identity*: Special flavor of principal-like entity
> > > > > representing a consumer relationship - “ExternalConsumer”
> > > > > *C. Map Containers to Consumers*: A way to specify letting a given
> > > > > consumer access a given sharing container - “Listing”
> > > > > *D. Consumer Endpoint*: A segregated API access-point for
> consumers -
> > > > > “EndpointConfig”
> > > > > *E. Objects in Containers*: A way to specify which in-catalog
> objects
> > > > > belong in the big sharing container - “ShareMembership”
> > > > >
> > > > > This still bakes in a bit of implementation detail since it begs
> the
> > > > > question why not just let core RBAC/grants handle (C) and (E). To
> > that
> > > > end,
> > > > > and addressing problem (2) of where the "form *is* function", I
> > added a
> > > > > "Concept Mapping" tab:
> > > > >
> > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > > >
> > > > > I think the details of more advanced features can still be hashed
> out
> > > > > incrementally without blocking work on an MVP (and certainly the
> MVP
> > > > scope
> > > > > could be trimmed so we can launch-and-iterate), so as long as
> there's
> > > > > community alignment on the general direction I propose we could
> move
> > > > > forward on basic building blocks. Or if building blocks themselves
> > are
> > > > > controversial we could still proceed with some prototyping across
> > > > different
> > > > > options.
> > > > >
> > > > > Cheers,
> > > > > Dennis
> > > > >
> > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <
> > jb@nanthrax.net>
> > > > > wrote:
> > > > >
> > > > >> Hi everyone,
> > > > >>
> > > > >> I would like to resume the discussion about Data Sharing in
> Polaris
> > > > >> and propose focusing on three major items to help us move forward.
> > > > >>
> > > > >> 1. What is data sharing?
> > > > >> In a nutshell, it is an open protocol proposal allowing
> > > > >> organizations to share live data without copying or replication.
> The
> > > > >> main goal is zero-copy sharing achieved through shared metadata
> and
> > > > >> authorization.
> > > > >>
> > > > >> On Polaris, this would mean:
> > > > >> a. The consumer is authenticated as usual.
> > > > >> b. The Polaris server provides the consumer with direct access to
> > the
> > > > >> data files (e.g., Parquet, as specified in the metadata.json).
> > > > >> c. The consumer reads the data files directly using their
> preferred
> > > > >> tool (e.g., a query engine).
> > > > >>
> > > > >> While Polaris already has most of the necessary underlying
> > > > >> capabilities, we need clear branding/packaging around this use
> case.
> > > > >> Specifically, this means:
> > > > >> a. A share is a dedicated catalog in the Polaris server, or a
> > catalog
> > > > >> role scoped to specific namespaces.
> > > > >> b. A recipient is represented by a principal and a principal role.
> > > > >> c. A consumer profile consists of the catalog URI and the OAuth2
> > > > >> client_id/client_secret.
> > > > >> d. We use credential vending (temporary STS credentials) for
> > consumer
> > > > >> access to the data files.
> > > > >> e. We use the Polaris IRC layer to expose the metadata and the
> file
> > > > >> locations/access.
> > > > >> f. We can also support Polaris-to-Polaris sharing via Federation.
> > > > >>
> > > > >> Concretely:
> > > > >>
> > > > >> - On the provider side, we use the standard Polaris RBAC model
> > > > >> (assigning grants to catalog roles, and assigning those catalog
> > roles
> > > > >> to principal roles):
> > > > >> - Create a dedicated principal with its own
> > > > client_id/client_secret.
> > > > >> - Create a catalog role with TABLE_READ_DATA, TABLE_LIST,
> and
> > > > >> NAMESPACE_LIST grants on the namespaces to be shared (no write
> > > > >> grants).
> > > > >> - Create a principal role per destination, attach the
> catalog
> > > > >> role, and assign the principal.
> > > > >> - The destination engine uses loadTable with the
> > > > >> "X-Iceberg-Access-Delegation: vended-credentials" header. Polaris
> > then
> > > > >> emits temporary, read-only storage credentials, allowing the
> engine
> > to
> > > > >> read the data files directly from the owner's bucket without
> > accessing
> > > > >> the rest of the bucket. This achieves zero-copy sharing.
> > > > >> - On the recipient/consumer side, no special configuration is
> > > > >> needed; any IRC client can use it.
> > > > >>
> > > > >> 2. What is data sharing not?
> > > > >> I previously considered adding an Arrow Flight endpoint to
> > > > >> Polaris, but I no longer think it is a good idea for the initial
> > > > >> phase. An Arrow Flight endpoint would serve the data directly,
> which
> > > > >> would require Polaris to read the data and expose it. I am against
> > > > >> embedding a query engine in Polaris, as this tight coupling feels
> > like
> > > > >> an anti-pattern. While we could use a delegation service to serve
> > the
> > > > >> Flight endpoint, I think it should be considered for a later
> phase,
> > if
> > > > >> at all, since a Flight endpoint might be better suited as a
> consumer
> > > > >> responsibility and could encourage data copying.
> > > > >>
> > > > >> The first phase should focus on packaging and branding the
> > > > >> capabilities we already have.
> > > > >>
> > > > >> 3. What could the Data Sharing "package" look like in Polaris?
> > > > >> To make Data Sharing obvious and easy to use, we should
> > introduce
> > > > >> the missing concept of a "share"—a named definition that users can
> > > > >> list, discover, and define. The goal is to materialize the user's
> > > > >> intent ("I want to share these tables with this consumer until
> this
> > > > >> date") into standard Polaris entities (principal, principal role,
> > > > >> catalog role, and grants).
> > > > >>
> > > > >> Therefore, I propose adding a /shares endpoint to the Polaris
> > > > >> Management API to package and materialize these share definitions
> > into
> > > > >> Polaris entities. This would provide a usable, first-version
> package
> > > > >> for Polaris Data Sharing.
> > > > >>
> > > > >> I hope this addresses Dmitri's questions and provides a more
> > concrete
> > > > >> scenario for Polaris Data Sharing.
> > > > >>
> > > > >> Thoughts?
> > > > >>
> > > > >> Regards,
> > > > >> JB
> > > > >>
> > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <dh...@apache.org>
> wrote:
> > > > >> >
> > > > >> > Hi All,
> > > > >> >
> > > > >> > One big emerging enterprise use case coming up as more people
> > > > >> consolidate
> > > > >> > Data Lakehouses and Catalogs is something commonly known as
> "Data
> > > > >> Sharing",
> > > > >> > an more specifically over the course of adoption of open table
> > formats
> > > > >> > "Open Data Sharing".
> > > > >> >
> > > > >> > Examples of existing managed service providers' Data Sharing
> > features:
> > > > >> >
> > > > >> > https://www.databricks.com/product/delta-sharing
> > > > >> > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > > >> >
> > > >
> https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > > >> >
> > > >
> https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > > >> >
> > > > >>
> > > >
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > > >> >
> > > > >> > The basic idea is that when you share data between different
> > > > companies,
> > > > >> you
> > > > >> > need a first-class governance/management layer and extra
> > > > >> bells-and-whistles
> > > > >> > that are distinct from just the basic capabilities of RBAC or
> > > > >> generalized
> > > > >> > access-control (i.e. if you're sharing across
> partially-untrusted
> > org
> > > > >> > boundaries, you don't just let the consumer organization log
> into
> > your
> > > > >> > datalake like one of your own employees).
> > > > >> >
> > > > >> > JB and I put together this high-level proposal for supporting
> Open
> > > > >> Sharing
> > > > >> > in Polaris:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > > >> >
> > > > >> > Tentatively, it means adding ~5 logical data model constructs,
> > some of
> > > > >> > which may be a first-class PolarisEntity type, others subtypes
> of
> > > > >> existing
> > > > >> > entities, and others just a nested construct:
> > > > >> >
> > > > >> > - ShareEntity (would behave similarly to a Catalog)
> > > > >> > - ExternalConsumer (mostly inherits from Principal)
> > > > >> > - Listing (Similar to a "role grant" but has different
> > metadata)
> > > > >> > - EndpointConfig (nested config under Listing)
> > > > >> > - ShareMembership (Similar to a "securable grant" but
> different
> > > > >> metadata)
> > > > >> >
> > > > >> > Feedback/comments welcome! I'll also bring it up for live
> > discussion
> > > > if
> > > > >> > there's time in the community sync.
> > > > >> >
> > > > >> > Cheers,
> > > > >> > Dennis
> > > > >>
> > > > >
> > > >
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dennis Huo <dh...@apache.org>.
As discussed live, looks like we both might have some drafts we can compare
and choose/merge from :)
Here's my current draft spec PR; it opts for my preferred syntax choices
for now (/api/management/ for share-management, /api/shares/ for consumer
data-plane): https://github.com/apache/polaris/pull/5446
Since we were talking about preferring mdfiles for design iteration here's
a companion design-addendum which just enumerates what the end-to-end user
journeys look like syntactically for each combination of design choices:
https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md
Please let me know if anyone prefers a Google Doc version of the
design-addendum for easier commenting, or if the design-addendum itself
should also be modeled as a PR on some free-floating branch just to capture
comments.
On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi everyone,
>
> I am resuming work on this proposal.
>
> Regarding our recent discussion, I will move forward with opening a
> draft PR for the /api/shares/v1 endpoint to help illustrate the
> design.
>
> I will update this thread once the PR is ready.
>
> Thanks,
> JB
>
> On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
> >
> > Hi Dennis,
> >
> > Thanks for the recap!
> >
> > I was thinking that share management is probably distinct from catalog
> > management. A user who already has a realm+catalog may have an option to
> > manage shares, but not the catalog itself. Would it make sense to
> separate
> > the share management API from the general Polaris management API? I mean
> > using a different prefix for URIs.
> >
> > For example: /api/shares/v1/
> >
> > Obviously clients should not make any assumptions about the base path
> > (/api/shares), which could be different in different deployments (e.g.
> > going through a gateway). Clients must be able to take the base URI as a
> > config option. Path segments after /v1/ will be specified by Polaris
> (Open
> > API yaml).
> >
> > WDYT?
> >
> > Thanks,
> > Dmitri.
> >
> > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <dh...@apache.org> wrote:
> >
> > > Just to recap some of the discussion from today's community sync,
> Dmitri,
> > > Robert, and Alex helped raise a few considerations to roll into the
> next
> > > design/prototyping phase:
> > >
> > > 1. Given the way sharing is packaged as a separate thing, it may not be
> > > obvious to users that historical snapshots are equally available to
> > > consumers as the latest snapshot (by virtue of sharing via the
> standard IRC
> > > protocol) - we should make sure to document that fact well, including
> the
> > > pitfall that deleting rows by updating a table doesn't necessarily mean
> > > those rows become inaccessible to consumers
> > > 2. Make sure not to hard-code assumptions about the existence of an
> > > internal PrincipalEntity backing an ExternalConsumer - even if the
> first
> > > MVP implementation uses a nested/hidden Principal of some type that
> holds a
> > > Polaris-managed client_id/client_secret, the 3rd-party IdP use cases
> > > require "implicit" Principals resolved via the oauth token. In such
> cases
> > > the ExternalConsumer contains the metadata coordinating the three-way
> > > handshake between the data-provider, data-consumer, and IdP, but there
> is
> > > no PrincipalEntity underlying it.
> > > 3. Longer-term, we may want to tackle "on-behalf-of" semantics -
> initially,
> > > the way an ExternalConsumer works is that the finer-grained consumer
> users
> > > share a single ExternalConsumer identity, just like how a FEDERATED
> catalog
> > > shares a single service principal in its ConnectionConfig. But in some
> > > cases the consumer may want some end-to-end attribution of fine-grained
> > > users, like "on-behalf-of: bob @ consumercompany.com" when the
> > > ExternalConsumer implicit principal is something like
> > > "externalconsumer$consumercompany"
> > > 4. We can iterate more rapidly on a draft spec PR and also expand on
> the
> > > detailed technical design
> > >
> > > Cheers,
> > > Dennis
> > >
> > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <dh...@apache.org> wrote:
> > >
> > > > Thanks JB for reviving this thread, and belated thanks to all
> reviewers
> > > > who have left feedback in the doc or this thread so far! And
> apologies
> > > for
> > > > the silence on this, it's still been on my radar but I got
> distracted by
> > > > some other work for awhile.
> > > >
> > > > Echoing JB's summary, the proposal is indeed that we're mostly just
> > > > reusing existing Authz/RBAC machinery, and the translation layer to
> the
> > > new
> > > > sharing concepts sits *on top of* the existing authz engine as much
> as
> > > > possible.
> > > >
> > > > So even if we're adding layers of indirection on
> ExternalConsumer/Listing
> > > > -> Principal/PrincipalRole/CatalogRole+grants, the parts that end up
> > > doing
> > > > the underlying authz just reuse existing Authorizer logic.
> > > >
> > > > Based on some other offline discussion with Dmitri, I realized why
> the
> > > > discussion around "requirements" may have been confusing:
> > > >
> > > > 1. My attempts to shorten the doc inadvertently cut out some of the
> more
> > > > "abstract requirements" formulations
> > > > 2. The project in itself is somewhat more of a "form *as* function"
> > > > feature than *strictly* functional in nature -- for example, many
> > > correctly
> > > > pointed out that "sharing" via the existing RBAC/Principals syntax is
> > > > already *possible* just by letting a cross-org "consumer" be a
> > > first-class
> > > > principal and access your own company's main Polaris catalog
> endpoints.
> > > >
> > > > For (1) I added this to the Requirements section in the doc to better
> > > > clarify the abstract form of the data model element:
> > > >
> > > > *A. Sharing Container*: Container concept for holding things you
> want to
> > > > share - “ShareEntity”
> > > > *B. Consumer Identity*: Special flavor of principal-like entity
> > > > representing a consumer relationship - “ExternalConsumer”
> > > > *C. Map Containers to Consumers*: A way to specify letting a given
> > > > consumer access a given sharing container - “Listing”
> > > > *D. Consumer Endpoint*: A segregated API access-point for consumers -
> > > > “EndpointConfig”
> > > > *E. Objects in Containers*: A way to specify which in-catalog objects
> > > > belong in the big sharing container - “ShareMembership”
> > > >
> > > > This still bakes in a bit of implementation detail since it begs the
> > > > question why not just let core RBAC/grants handle (C) and (E). To
> that
> > > end,
> > > > and addressing problem (2) of where the "form *is* function", I
> added a
> > > > "Concept Mapping" tab:
> > > >
> > > >
> > > >
> > >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > > >
> > > > I think the details of more advanced features can still be hashed out
> > > > incrementally without blocking work on an MVP (and certainly the MVP
> > > scope
> > > > could be trimmed so we can launch-and-iterate), so as long as there's
> > > > community alignment on the general direction I propose we could move
> > > > forward on basic building blocks. Or if building blocks themselves
> are
> > > > controversial we could still proceed with some prototyping across
> > > different
> > > > options.
> > > >
> > > > Cheers,
> > > > Dennis
> > > >
> > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > > > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> I would like to resume the discussion about Data Sharing in Polaris
> > > >> and propose focusing on three major items to help us move forward.
> > > >>
> > > >> 1. What is data sharing?
> > > >> In a nutshell, it is an open protocol proposal allowing
> > > >> organizations to share live data without copying or replication. The
> > > >> main goal is zero-copy sharing achieved through shared metadata and
> > > >> authorization.
> > > >>
> > > >> On Polaris, this would mean:
> > > >> a. The consumer is authenticated as usual.
> > > >> b. The Polaris server provides the consumer with direct access to
> the
> > > >> data files (e.g., Parquet, as specified in the metadata.json).
> > > >> c. The consumer reads the data files directly using their preferred
> > > >> tool (e.g., a query engine).
> > > >>
> > > >> While Polaris already has most of the necessary underlying
> > > >> capabilities, we need clear branding/packaging around this use case.
> > > >> Specifically, this means:
> > > >> a. A share is a dedicated catalog in the Polaris server, or a
> catalog
> > > >> role scoped to specific namespaces.
> > > >> b. A recipient is represented by a principal and a principal role.
> > > >> c. A consumer profile consists of the catalog URI and the OAuth2
> > > >> client_id/client_secret.
> > > >> d. We use credential vending (temporary STS credentials) for
> consumer
> > > >> access to the data files.
> > > >> e. We use the Polaris IRC layer to expose the metadata and the file
> > > >> locations/access.
> > > >> f. We can also support Polaris-to-Polaris sharing via Federation.
> > > >>
> > > >> Concretely:
> > > >>
> > > >> - On the provider side, we use the standard Polaris RBAC model
> > > >> (assigning grants to catalog roles, and assigning those catalog
> roles
> > > >> to principal roles):
> > > >> - Create a dedicated principal with its own
> > > client_id/client_secret.
> > > >> - Create a catalog role with TABLE_READ_DATA, TABLE_LIST, and
> > > >> NAMESPACE_LIST grants on the namespaces to be shared (no write
> > > >> grants).
> > > >> - Create a principal role per destination, attach the catalog
> > > >> role, and assign the principal.
> > > >> - The destination engine uses loadTable with the
> > > >> "X-Iceberg-Access-Delegation: vended-credentials" header. Polaris
> then
> > > >> emits temporary, read-only storage credentials, allowing the engine
> to
> > > >> read the data files directly from the owner's bucket without
> accessing
> > > >> the rest of the bucket. This achieves zero-copy sharing.
> > > >> - On the recipient/consumer side, no special configuration is
> > > >> needed; any IRC client can use it.
> > > >>
> > > >> 2. What is data sharing not?
> > > >> I previously considered adding an Arrow Flight endpoint to
> > > >> Polaris, but I no longer think it is a good idea for the initial
> > > >> phase. An Arrow Flight endpoint would serve the data directly, which
> > > >> would require Polaris to read the data and expose it. I am against
> > > >> embedding a query engine in Polaris, as this tight coupling feels
> like
> > > >> an anti-pattern. While we could use a delegation service to serve
> the
> > > >> Flight endpoint, I think it should be considered for a later phase,
> if
> > > >> at all, since a Flight endpoint might be better suited as a consumer
> > > >> responsibility and could encourage data copying.
> > > >>
> > > >> The first phase should focus on packaging and branding the
> > > >> capabilities we already have.
> > > >>
> > > >> 3. What could the Data Sharing "package" look like in Polaris?
> > > >> To make Data Sharing obvious and easy to use, we should
> introduce
> > > >> the missing concept of a "share"—a named definition that users can
> > > >> list, discover, and define. The goal is to materialize the user's
> > > >> intent ("I want to share these tables with this consumer until this
> > > >> date") into standard Polaris entities (principal, principal role,
> > > >> catalog role, and grants).
> > > >>
> > > >> Therefore, I propose adding a /shares endpoint to the Polaris
> > > >> Management API to package and materialize these share definitions
> into
> > > >> Polaris entities. This would provide a usable, first-version package
> > > >> for Polaris Data Sharing.
> > > >>
> > > >> I hope this addresses Dmitri's questions and provides a more
> concrete
> > > >> scenario for Polaris Data Sharing.
> > > >>
> > > >> Thoughts?
> > > >>
> > > >> Regards,
> > > >> JB
> > > >>
> > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <dh...@apache.org> wrote:
> > > >> >
> > > >> > Hi All,
> > > >> >
> > > >> > One big emerging enterprise use case coming up as more people
> > > >> consolidate
> > > >> > Data Lakehouses and Catalogs is something commonly known as "Data
> > > >> Sharing",
> > > >> > an more specifically over the course of adoption of open table
> formats
> > > >> > "Open Data Sharing".
> > > >> >
> > > >> > Examples of existing managed service providers' Data Sharing
> features:
> > > >> >
> > > >> > https://www.databricks.com/product/delta-sharing
> > > >> > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > > >> >
> > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > > >> >
> > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > > >> >
> > > >>
> > >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > > >> >
> > > >> > The basic idea is that when you share data between different
> > > companies,
> > > >> you
> > > >> > need a first-class governance/management layer and extra
> > > >> bells-and-whistles
> > > >> > that are distinct from just the basic capabilities of RBAC or
> > > >> generalized
> > > >> > access-control (i.e. if you're sharing across partially-untrusted
> org
> > > >> > boundaries, you don't just let the consumer organization log into
> your
> > > >> > datalake like one of your own employees).
> > > >> >
> > > >> > JB and I put together this high-level proposal for supporting Open
> > > >> Sharing
> > > >> > in Polaris:
> > > >> >
> > > >> >
> > > >>
> > >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > > >> >
> > > >> > Tentatively, it means adding ~5 logical data model constructs,
> some of
> > > >> > which may be a first-class PolarisEntity type, others subtypes of
> > > >> existing
> > > >> > entities, and others just a nested construct:
> > > >> >
> > > >> > - ShareEntity (would behave similarly to a Catalog)
> > > >> > - ExternalConsumer (mostly inherits from Principal)
> > > >> > - Listing (Similar to a "role grant" but has different
> metadata)
> > > >> > - EndpointConfig (nested config under Listing)
> > > >> > - ShareMembership (Similar to a "securable grant" but different
> > > >> metadata)
> > > >> >
> > > >> > Feedback/comments welcome! I'll also bring it up for live
> discussion
> > > if
> > > >> > there's time in the community sync.
> > > >> >
> > > >> > Cheers,
> > > >> > Dennis
> > > >>
> > > >
> > >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi everyone,
I am resuming work on this proposal.
Regarding our recent discussion, I will move forward with opening a
draft PR for the /api/shares/v1 endpoint to help illustrate the
design.
I will update this thread once the PR is ready.
Thanks,
JB
On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <di...@apache.org> wrote:
>
> Hi Dennis,
>
> Thanks for the recap!
>
> I was thinking that share management is probably distinct from catalog
> management. A user who already has a realm+catalog may have an option to
> manage shares, but not the catalog itself. Would it make sense to separate
> the share management API from the general Polaris management API? I mean
> using a different prefix for URIs.
>
> For example: /api/shares/v1/
>
> Obviously clients should not make any assumptions about the base path
> (/api/shares), which could be different in different deployments (e.g.
> going through a gateway). Clients must be able to take the base URI as a
> config option. Path segments after /v1/ will be specified by Polaris (Open
> API yaml).
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <dh...@apache.org> wrote:
>
> > Just to recap some of the discussion from today's community sync, Dmitri,
> > Robert, and Alex helped raise a few considerations to roll into the next
> > design/prototyping phase:
> >
> > 1. Given the way sharing is packaged as a separate thing, it may not be
> > obvious to users that historical snapshots are equally available to
> > consumers as the latest snapshot (by virtue of sharing via the standard IRC
> > protocol) - we should make sure to document that fact well, including the
> > pitfall that deleting rows by updating a table doesn't necessarily mean
> > those rows become inaccessible to consumers
> > 2. Make sure not to hard-code assumptions about the existence of an
> > internal PrincipalEntity backing an ExternalConsumer - even if the first
> > MVP implementation uses a nested/hidden Principal of some type that holds a
> > Polaris-managed client_id/client_secret, the 3rd-party IdP use cases
> > require "implicit" Principals resolved via the oauth token. In such cases
> > the ExternalConsumer contains the metadata coordinating the three-way
> > handshake between the data-provider, data-consumer, and IdP, but there is
> > no PrincipalEntity underlying it.
> > 3. Longer-term, we may want to tackle "on-behalf-of" semantics - initially,
> > the way an ExternalConsumer works is that the finer-grained consumer users
> > share a single ExternalConsumer identity, just like how a FEDERATED catalog
> > shares a single service principal in its ConnectionConfig. But in some
> > cases the consumer may want some end-to-end attribution of fine-grained
> > users, like "on-behalf-of: bob @ consumercompany.com" when the
> > ExternalConsumer implicit principal is something like
> > "externalconsumer$consumercompany"
> > 4. We can iterate more rapidly on a draft spec PR and also expand on the
> > detailed technical design
> >
> > Cheers,
> > Dennis
> >
> > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <dh...@apache.org> wrote:
> >
> > > Thanks JB for reviving this thread, and belated thanks to all reviewers
> > > who have left feedback in the doc or this thread so far! And apologies
> > for
> > > the silence on this, it's still been on my radar but I got distracted by
> > > some other work for awhile.
> > >
> > > Echoing JB's summary, the proposal is indeed that we're mostly just
> > > reusing existing Authz/RBAC machinery, and the translation layer to the
> > new
> > > sharing concepts sits *on top of* the existing authz engine as much as
> > > possible.
> > >
> > > So even if we're adding layers of indirection on ExternalConsumer/Listing
> > > -> Principal/PrincipalRole/CatalogRole+grants, the parts that end up
> > doing
> > > the underlying authz just reuse existing Authorizer logic.
> > >
> > > Based on some other offline discussion with Dmitri, I realized why the
> > > discussion around "requirements" may have been confusing:
> > >
> > > 1. My attempts to shorten the doc inadvertently cut out some of the more
> > > "abstract requirements" formulations
> > > 2. The project in itself is somewhat more of a "form *as* function"
> > > feature than *strictly* functional in nature -- for example, many
> > correctly
> > > pointed out that "sharing" via the existing RBAC/Principals syntax is
> > > already *possible* just by letting a cross-org "consumer" be a
> > first-class
> > > principal and access your own company's main Polaris catalog endpoints.
> > >
> > > For (1) I added this to the Requirements section in the doc to better
> > > clarify the abstract form of the data model element:
> > >
> > > *A. Sharing Container*: Container concept for holding things you want to
> > > share - “ShareEntity”
> > > *B. Consumer Identity*: Special flavor of principal-like entity
> > > representing a consumer relationship - “ExternalConsumer”
> > > *C. Map Containers to Consumers*: A way to specify letting a given
> > > consumer access a given sharing container - “Listing”
> > > *D. Consumer Endpoint*: A segregated API access-point for consumers -
> > > “EndpointConfig”
> > > *E. Objects in Containers*: A way to specify which in-catalog objects
> > > belong in the big sharing container - “ShareMembership”
> > >
> > > This still bakes in a bit of implementation detail since it begs the
> > > question why not just let core RBAC/grants handle (C) and (E). To that
> > end,
> > > and addressing problem (2) of where the "form *is* function", I added a
> > > "Concept Mapping" tab:
> > >
> > >
> > >
> > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> > >
> > > I think the details of more advanced features can still be hashed out
> > > incrementally without blocking work on an MVP (and certainly the MVP
> > scope
> > > could be trimmed so we can launch-and-iterate), so as long as there's
> > > community alignment on the general direction I propose we could move
> > > forward on basic building blocks. Or if building blocks themselves are
> > > controversial we could still proceed with some prototyping across
> > different
> > > options.
> > >
> > > Cheers,
> > > Dennis
> > >
> > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> I would like to resume the discussion about Data Sharing in Polaris
> > >> and propose focusing on three major items to help us move forward.
> > >>
> > >> 1. What is data sharing?
> > >> In a nutshell, it is an open protocol proposal allowing
> > >> organizations to share live data without copying or replication. The
> > >> main goal is zero-copy sharing achieved through shared metadata and
> > >> authorization.
> > >>
> > >> On Polaris, this would mean:
> > >> a. The consumer is authenticated as usual.
> > >> b. The Polaris server provides the consumer with direct access to the
> > >> data files (e.g., Parquet, as specified in the metadata.json).
> > >> c. The consumer reads the data files directly using their preferred
> > >> tool (e.g., a query engine).
> > >>
> > >> While Polaris already has most of the necessary underlying
> > >> capabilities, we need clear branding/packaging around this use case.
> > >> Specifically, this means:
> > >> a. A share is a dedicated catalog in the Polaris server, or a catalog
> > >> role scoped to specific namespaces.
> > >> b. A recipient is represented by a principal and a principal role.
> > >> c. A consumer profile consists of the catalog URI and the OAuth2
> > >> client_id/client_secret.
> > >> d. We use credential vending (temporary STS credentials) for consumer
> > >> access to the data files.
> > >> e. We use the Polaris IRC layer to expose the metadata and the file
> > >> locations/access.
> > >> f. We can also support Polaris-to-Polaris sharing via Federation.
> > >>
> > >> Concretely:
> > >>
> > >> - On the provider side, we use the standard Polaris RBAC model
> > >> (assigning grants to catalog roles, and assigning those catalog roles
> > >> to principal roles):
> > >> - Create a dedicated principal with its own
> > client_id/client_secret.
> > >> - Create a catalog role with TABLE_READ_DATA, TABLE_LIST, and
> > >> NAMESPACE_LIST grants on the namespaces to be shared (no write
> > >> grants).
> > >> - Create a principal role per destination, attach the catalog
> > >> role, and assign the principal.
> > >> - The destination engine uses loadTable with the
> > >> "X-Iceberg-Access-Delegation: vended-credentials" header. Polaris then
> > >> emits temporary, read-only storage credentials, allowing the engine to
> > >> read the data files directly from the owner's bucket without accessing
> > >> the rest of the bucket. This achieves zero-copy sharing.
> > >> - On the recipient/consumer side, no special configuration is
> > >> needed; any IRC client can use it.
> > >>
> > >> 2. What is data sharing not?
> > >> I previously considered adding an Arrow Flight endpoint to
> > >> Polaris, but I no longer think it is a good idea for the initial
> > >> phase. An Arrow Flight endpoint would serve the data directly, which
> > >> would require Polaris to read the data and expose it. I am against
> > >> embedding a query engine in Polaris, as this tight coupling feels like
> > >> an anti-pattern. While we could use a delegation service to serve the
> > >> Flight endpoint, I think it should be considered for a later phase, if
> > >> at all, since a Flight endpoint might be better suited as a consumer
> > >> responsibility and could encourage data copying.
> > >>
> > >> The first phase should focus on packaging and branding the
> > >> capabilities we already have.
> > >>
> > >> 3. What could the Data Sharing "package" look like in Polaris?
> > >> To make Data Sharing obvious and easy to use, we should introduce
> > >> the missing concept of a "share"—a named definition that users can
> > >> list, discover, and define. The goal is to materialize the user's
> > >> intent ("I want to share these tables with this consumer until this
> > >> date") into standard Polaris entities (principal, principal role,
> > >> catalog role, and grants).
> > >>
> > >> Therefore, I propose adding a /shares endpoint to the Polaris
> > >> Management API to package and materialize these share definitions into
> > >> Polaris entities. This would provide a usable, first-version package
> > >> for Polaris Data Sharing.
> > >>
> > >> I hope this addresses Dmitri's questions and provides a more concrete
> > >> scenario for Polaris Data Sharing.
> > >>
> > >> Thoughts?
> > >>
> > >> Regards,
> > >> JB
> > >>
> > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <dh...@apache.org> wrote:
> > >> >
> > >> > Hi All,
> > >> >
> > >> > One big emerging enterprise use case coming up as more people
> > >> consolidate
> > >> > Data Lakehouses and Catalogs is something commonly known as "Data
> > >> Sharing",
> > >> > an more specifically over the course of adoption of open table formats
> > >> > "Open Data Sharing".
> > >> >
> > >> > Examples of existing managed service providers' Data Sharing features:
> > >> >
> > >> > https://www.databricks.com/product/delta-sharing
> > >> > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > >> >
> > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > >> >
> > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> > >> >
> > >>
> > https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> > >> >
> > >> > The basic idea is that when you share data between different
> > companies,
> > >> you
> > >> > need a first-class governance/management layer and extra
> > >> bells-and-whistles
> > >> > that are distinct from just the basic capabilities of RBAC or
> > >> generalized
> > >> > access-control (i.e. if you're sharing across partially-untrusted org
> > >> > boundaries, you don't just let the consumer organization log into your
> > >> > datalake like one of your own employees).
> > >> >
> > >> > JB and I put together this high-level proposal for supporting Open
> > >> Sharing
> > >> > in Polaris:
> > >> >
> > >> >
> > >>
> > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> > >> >
> > >> > Tentatively, it means adding ~5 logical data model constructs, some of
> > >> > which may be a first-class PolarisEntity type, others subtypes of
> > >> existing
> > >> > entities, and others just a nested construct:
> > >> >
> > >> > - ShareEntity (would behave similarly to a Catalog)
> > >> > - ExternalConsumer (mostly inherits from Principal)
> > >> > - Listing (Similar to a "role grant" but has different metadata)
> > >> > - EndpointConfig (nested config under Listing)
> > >> > - ShareMembership (Similar to a "securable grant" but different
> > >> metadata)
> > >> >
> > >> > Feedback/comments welcome! I'll also bring it up for live discussion
> > if
> > >> > there's time in the community sync.
> > >> >
> > >> > Cheers,
> > >> > Dennis
> > >>
> > >
> >
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Dennis,
Thanks for the recap!
I was thinking that share management is probably distinct from catalog
management. A user who already has a realm+catalog may have an option to
manage shares, but not the catalog itself. Would it make sense to separate
the share management API from the general Polaris management API? I mean
using a different prefix for URIs.
For example: /api/shares/v1/
Obviously clients should not make any assumptions about the base path
(/api/shares), which could be different in different deployments (e.g.
going through a gateway). Clients must be able to take the base URI as a
config option. Path segments after /v1/ will be specified by Polaris (Open
API yaml).
WDYT?
Thanks,
Dmitri.
On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <dh...@apache.org> wrote:
> Just to recap some of the discussion from today's community sync, Dmitri,
> Robert, and Alex helped raise a few considerations to roll into the next
> design/prototyping phase:
>
> 1. Given the way sharing is packaged as a separate thing, it may not be
> obvious to users that historical snapshots are equally available to
> consumers as the latest snapshot (by virtue of sharing via the standard IRC
> protocol) - we should make sure to document that fact well, including the
> pitfall that deleting rows by updating a table doesn't necessarily mean
> those rows become inaccessible to consumers
> 2. Make sure not to hard-code assumptions about the existence of an
> internal PrincipalEntity backing an ExternalConsumer - even if the first
> MVP implementation uses a nested/hidden Principal of some type that holds a
> Polaris-managed client_id/client_secret, the 3rd-party IdP use cases
> require "implicit" Principals resolved via the oauth token. In such cases
> the ExternalConsumer contains the metadata coordinating the three-way
> handshake between the data-provider, data-consumer, and IdP, but there is
> no PrincipalEntity underlying it.
> 3. Longer-term, we may want to tackle "on-behalf-of" semantics - initially,
> the way an ExternalConsumer works is that the finer-grained consumer users
> share a single ExternalConsumer identity, just like how a FEDERATED catalog
> shares a single service principal in its ConnectionConfig. But in some
> cases the consumer may want some end-to-end attribution of fine-grained
> users, like "on-behalf-of: bob @ consumercompany.com" when the
> ExternalConsumer implicit principal is something like
> "externalconsumer$consumercompany"
> 4. We can iterate more rapidly on a draft spec PR and also expand on the
> detailed technical design
>
> Cheers,
> Dennis
>
> On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <dh...@apache.org> wrote:
>
> > Thanks JB for reviving this thread, and belated thanks to all reviewers
> > who have left feedback in the doc or this thread so far! And apologies
> for
> > the silence on this, it's still been on my radar but I got distracted by
> > some other work for awhile.
> >
> > Echoing JB's summary, the proposal is indeed that we're mostly just
> > reusing existing Authz/RBAC machinery, and the translation layer to the
> new
> > sharing concepts sits *on top of* the existing authz engine as much as
> > possible.
> >
> > So even if we're adding layers of indirection on ExternalConsumer/Listing
> > -> Principal/PrincipalRole/CatalogRole+grants, the parts that end up
> doing
> > the underlying authz just reuse existing Authorizer logic.
> >
> > Based on some other offline discussion with Dmitri, I realized why the
> > discussion around "requirements" may have been confusing:
> >
> > 1. My attempts to shorten the doc inadvertently cut out some of the more
> > "abstract requirements" formulations
> > 2. The project in itself is somewhat more of a "form *as* function"
> > feature than *strictly* functional in nature -- for example, many
> correctly
> > pointed out that "sharing" via the existing RBAC/Principals syntax is
> > already *possible* just by letting a cross-org "consumer" be a
> first-class
> > principal and access your own company's main Polaris catalog endpoints.
> >
> > For (1) I added this to the Requirements section in the doc to better
> > clarify the abstract form of the data model element:
> >
> > *A. Sharing Container*: Container concept for holding things you want to
> > share - “ShareEntity”
> > *B. Consumer Identity*: Special flavor of principal-like entity
> > representing a consumer relationship - “ExternalConsumer”
> > *C. Map Containers to Consumers*: A way to specify letting a given
> > consumer access a given sharing container - “Listing”
> > *D. Consumer Endpoint*: A segregated API access-point for consumers -
> > “EndpointConfig”
> > *E. Objects in Containers*: A way to specify which in-catalog objects
> > belong in the big sharing container - “ShareMembership”
> >
> > This still bakes in a bit of implementation detail since it begs the
> > question why not just let core RBAC/grants handle (C) and (E). To that
> end,
> > and addressing problem (2) of where the "form *is* function", I added a
> > "Concept Mapping" tab:
> >
> >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
> >
> > I think the details of more advanced features can still be hashed out
> > incrementally without blocking work on an MVP (and certainly the MVP
> scope
> > could be trimmed so we can launch-and-iterate), so as long as there's
> > community alignment on the general direction I propose we could move
> > forward on basic building blocks. Or if building blocks themselves are
> > controversial we could still proceed with some prototyping across
> different
> > options.
> >
> > Cheers,
> > Dennis
> >
> > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> >> Hi everyone,
> >>
> >> I would like to resume the discussion about Data Sharing in Polaris
> >> and propose focusing on three major items to help us move forward.
> >>
> >> 1. What is data sharing?
> >> In a nutshell, it is an open protocol proposal allowing
> >> organizations to share live data without copying or replication. The
> >> main goal is zero-copy sharing achieved through shared metadata and
> >> authorization.
> >>
> >> On Polaris, this would mean:
> >> a. The consumer is authenticated as usual.
> >> b. The Polaris server provides the consumer with direct access to the
> >> data files (e.g., Parquet, as specified in the metadata.json).
> >> c. The consumer reads the data files directly using their preferred
> >> tool (e.g., a query engine).
> >>
> >> While Polaris already has most of the necessary underlying
> >> capabilities, we need clear branding/packaging around this use case.
> >> Specifically, this means:
> >> a. A share is a dedicated catalog in the Polaris server, or a catalog
> >> role scoped to specific namespaces.
> >> b. A recipient is represented by a principal and a principal role.
> >> c. A consumer profile consists of the catalog URI and the OAuth2
> >> client_id/client_secret.
> >> d. We use credential vending (temporary STS credentials) for consumer
> >> access to the data files.
> >> e. We use the Polaris IRC layer to expose the metadata and the file
> >> locations/access.
> >> f. We can also support Polaris-to-Polaris sharing via Federation.
> >>
> >> Concretely:
> >>
> >> - On the provider side, we use the standard Polaris RBAC model
> >> (assigning grants to catalog roles, and assigning those catalog roles
> >> to principal roles):
> >> - Create a dedicated principal with its own
> client_id/client_secret.
> >> - Create a catalog role with TABLE_READ_DATA, TABLE_LIST, and
> >> NAMESPACE_LIST grants on the namespaces to be shared (no write
> >> grants).
> >> - Create a principal role per destination, attach the catalog
> >> role, and assign the principal.
> >> - The destination engine uses loadTable with the
> >> "X-Iceberg-Access-Delegation: vended-credentials" header. Polaris then
> >> emits temporary, read-only storage credentials, allowing the engine to
> >> read the data files directly from the owner's bucket without accessing
> >> the rest of the bucket. This achieves zero-copy sharing.
> >> - On the recipient/consumer side, no special configuration is
> >> needed; any IRC client can use it.
> >>
> >> 2. What is data sharing not?
> >> I previously considered adding an Arrow Flight endpoint to
> >> Polaris, but I no longer think it is a good idea for the initial
> >> phase. An Arrow Flight endpoint would serve the data directly, which
> >> would require Polaris to read the data and expose it. I am against
> >> embedding a query engine in Polaris, as this tight coupling feels like
> >> an anti-pattern. While we could use a delegation service to serve the
> >> Flight endpoint, I think it should be considered for a later phase, if
> >> at all, since a Flight endpoint might be better suited as a consumer
> >> responsibility and could encourage data copying.
> >>
> >> The first phase should focus on packaging and branding the
> >> capabilities we already have.
> >>
> >> 3. What could the Data Sharing "package" look like in Polaris?
> >> To make Data Sharing obvious and easy to use, we should introduce
> >> the missing concept of a "share"—a named definition that users can
> >> list, discover, and define. The goal is to materialize the user's
> >> intent ("I want to share these tables with this consumer until this
> >> date") into standard Polaris entities (principal, principal role,
> >> catalog role, and grants).
> >>
> >> Therefore, I propose adding a /shares endpoint to the Polaris
> >> Management API to package and materialize these share definitions into
> >> Polaris entities. This would provide a usable, first-version package
> >> for Polaris Data Sharing.
> >>
> >> I hope this addresses Dmitri's questions and provides a more concrete
> >> scenario for Polaris Data Sharing.
> >>
> >> Thoughts?
> >>
> >> Regards,
> >> JB
> >>
> >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <dh...@apache.org> wrote:
> >> >
> >> > Hi All,
> >> >
> >> > One big emerging enterprise use case coming up as more people
> >> consolidate
> >> > Data Lakehouses and Catalogs is something commonly known as "Data
> >> Sharing",
> >> > an more specifically over the course of adoption of open table formats
> >> > "Open Data Sharing".
> >> >
> >> > Examples of existing managed service providers' Data Sharing features:
> >> >
> >> > https://www.databricks.com/product/delta-sharing
> >> > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> >> >
> https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> >> >
> https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> >> >
> >>
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> >> >
> >> > The basic idea is that when you share data between different
> companies,
> >> you
> >> > need a first-class governance/management layer and extra
> >> bells-and-whistles
> >> > that are distinct from just the basic capabilities of RBAC or
> >> generalized
> >> > access-control (i.e. if you're sharing across partially-untrusted org
> >> > boundaries, you don't just let the consumer organization log into your
> >> > datalake like one of your own employees).
> >> >
> >> > JB and I put together this high-level proposal for supporting Open
> >> Sharing
> >> > in Polaris:
> >> >
> >> >
> >>
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> >> >
> >> > Tentatively, it means adding ~5 logical data model constructs, some of
> >> > which may be a first-class PolarisEntity type, others subtypes of
> >> existing
> >> > entities, and others just a nested construct:
> >> >
> >> > - ShareEntity (would behave similarly to a Catalog)
> >> > - ExternalConsumer (mostly inherits from Principal)
> >> > - Listing (Similar to a "role grant" but has different metadata)
> >> > - EndpointConfig (nested config under Listing)
> >> > - ShareMembership (Similar to a "securable grant" but different
> >> metadata)
> >> >
> >> > Feedback/comments welcome! I'll also bring it up for live discussion
> if
> >> > there's time in the community sync.
> >> >
> >> > Cheers,
> >> > Dennis
> >>
> >
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dennis Huo <dh...@apache.org>.
Just to recap some of the discussion from today's community sync, Dmitri,
Robert, and Alex helped raise a few considerations to roll into the next
design/prototyping phase:
1. Given the way sharing is packaged as a separate thing, it may not be
obvious to users that historical snapshots are equally available to
consumers as the latest snapshot (by virtue of sharing via the standard IRC
protocol) - we should make sure to document that fact well, including the
pitfall that deleting rows by updating a table doesn't necessarily mean
those rows become inaccessible to consumers
2. Make sure not to hard-code assumptions about the existence of an
internal PrincipalEntity backing an ExternalConsumer - even if the first
MVP implementation uses a nested/hidden Principal of some type that holds a
Polaris-managed client_id/client_secret, the 3rd-party IdP use cases
require "implicit" Principals resolved via the oauth token. In such cases
the ExternalConsumer contains the metadata coordinating the three-way
handshake between the data-provider, data-consumer, and IdP, but there is
no PrincipalEntity underlying it.
3. Longer-term, we may want to tackle "on-behalf-of" semantics - initially,
the way an ExternalConsumer works is that the finer-grained consumer users
share a single ExternalConsumer identity, just like how a FEDERATED catalog
shares a single service principal in its ConnectionConfig. But in some
cases the consumer may want some end-to-end attribution of fine-grained
users, like "on-behalf-of: bob @ consumercompany.com" when the
ExternalConsumer implicit principal is something like
"externalconsumer$consumercompany"
4. We can iterate more rapidly on a draft spec PR and also expand on the
detailed technical design
Cheers,
Dennis
On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <dh...@apache.org> wrote:
> Thanks JB for reviving this thread, and belated thanks to all reviewers
> who have left feedback in the doc or this thread so far! And apologies for
> the silence on this, it's still been on my radar but I got distracted by
> some other work for awhile.
>
> Echoing JB's summary, the proposal is indeed that we're mostly just
> reusing existing Authz/RBAC machinery, and the translation layer to the new
> sharing concepts sits *on top of* the existing authz engine as much as
> possible.
>
> So even if we're adding layers of indirection on ExternalConsumer/Listing
> -> Principal/PrincipalRole/CatalogRole+grants, the parts that end up doing
> the underlying authz just reuse existing Authorizer logic.
>
> Based on some other offline discussion with Dmitri, I realized why the
> discussion around "requirements" may have been confusing:
>
> 1. My attempts to shorten the doc inadvertently cut out some of the more
> "abstract requirements" formulations
> 2. The project in itself is somewhat more of a "form *as* function"
> feature than *strictly* functional in nature -- for example, many correctly
> pointed out that "sharing" via the existing RBAC/Principals syntax is
> already *possible* just by letting a cross-org "consumer" be a first-class
> principal and access your own company's main Polaris catalog endpoints.
>
> For (1) I added this to the Requirements section in the doc to better
> clarify the abstract form of the data model element:
>
> *A. Sharing Container*: Container concept for holding things you want to
> share - “ShareEntity”
> *B. Consumer Identity*: Special flavor of principal-like entity
> representing a consumer relationship - “ExternalConsumer”
> *C. Map Containers to Consumers*: A way to specify letting a given
> consumer access a given sharing container - “Listing”
> *D. Consumer Endpoint*: A segregated API access-point for consumers -
> “EndpointConfig”
> *E. Objects in Containers*: A way to specify which in-catalog objects
> belong in the big sharing container - “ShareMembership”
>
> This still bakes in a bit of implementation detail since it begs the
> question why not just let core RBAC/grants handle (C) and (E). To that end,
> and addressing problem (2) of where the "form *is* function", I added a
> "Concept Mapping" tab:
>
>
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
>
> I think the details of more advanced features can still be hashed out
> incrementally without blocking work on an MVP (and certainly the MVP scope
> could be trimmed so we can launch-and-iterate), so as long as there's
> community alignment on the general direction I propose we could move
> forward on basic building blocks. Or if building blocks themselves are
> controversial we could still proceed with some prototyping across different
> options.
>
> Cheers,
> Dennis
>
> On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi everyone,
>>
>> I would like to resume the discussion about Data Sharing in Polaris
>> and propose focusing on three major items to help us move forward.
>>
>> 1. What is data sharing?
>> In a nutshell, it is an open protocol proposal allowing
>> organizations to share live data without copying or replication. The
>> main goal is zero-copy sharing achieved through shared metadata and
>> authorization.
>>
>> On Polaris, this would mean:
>> a. The consumer is authenticated as usual.
>> b. The Polaris server provides the consumer with direct access to the
>> data files (e.g., Parquet, as specified in the metadata.json).
>> c. The consumer reads the data files directly using their preferred
>> tool (e.g., a query engine).
>>
>> While Polaris already has most of the necessary underlying
>> capabilities, we need clear branding/packaging around this use case.
>> Specifically, this means:
>> a. A share is a dedicated catalog in the Polaris server, or a catalog
>> role scoped to specific namespaces.
>> b. A recipient is represented by a principal and a principal role.
>> c. A consumer profile consists of the catalog URI and the OAuth2
>> client_id/client_secret.
>> d. We use credential vending (temporary STS credentials) for consumer
>> access to the data files.
>> e. We use the Polaris IRC layer to expose the metadata and the file
>> locations/access.
>> f. We can also support Polaris-to-Polaris sharing via Federation.
>>
>> Concretely:
>>
>> - On the provider side, we use the standard Polaris RBAC model
>> (assigning grants to catalog roles, and assigning those catalog roles
>> to principal roles):
>> - Create a dedicated principal with its own client_id/client_secret.
>> - Create a catalog role with TABLE_READ_DATA, TABLE_LIST, and
>> NAMESPACE_LIST grants on the namespaces to be shared (no write
>> grants).
>> - Create a principal role per destination, attach the catalog
>> role, and assign the principal.
>> - The destination engine uses loadTable with the
>> "X-Iceberg-Access-Delegation: vended-credentials" header. Polaris then
>> emits temporary, read-only storage credentials, allowing the engine to
>> read the data files directly from the owner's bucket without accessing
>> the rest of the bucket. This achieves zero-copy sharing.
>> - On the recipient/consumer side, no special configuration is
>> needed; any IRC client can use it.
>>
>> 2. What is data sharing not?
>> I previously considered adding an Arrow Flight endpoint to
>> Polaris, but I no longer think it is a good idea for the initial
>> phase. An Arrow Flight endpoint would serve the data directly, which
>> would require Polaris to read the data and expose it. I am against
>> embedding a query engine in Polaris, as this tight coupling feels like
>> an anti-pattern. While we could use a delegation service to serve the
>> Flight endpoint, I think it should be considered for a later phase, if
>> at all, since a Flight endpoint might be better suited as a consumer
>> responsibility and could encourage data copying.
>>
>> The first phase should focus on packaging and branding the
>> capabilities we already have.
>>
>> 3. What could the Data Sharing "package" look like in Polaris?
>> To make Data Sharing obvious and easy to use, we should introduce
>> the missing concept of a "share"—a named definition that users can
>> list, discover, and define. The goal is to materialize the user's
>> intent ("I want to share these tables with this consumer until this
>> date") into standard Polaris entities (principal, principal role,
>> catalog role, and grants).
>>
>> Therefore, I propose adding a /shares endpoint to the Polaris
>> Management API to package and materialize these share definitions into
>> Polaris entities. This would provide a usable, first-version package
>> for Polaris Data Sharing.
>>
>> I hope this addresses Dmitri's questions and provides a more concrete
>> scenario for Polaris Data Sharing.
>>
>> Thoughts?
>>
>> Regards,
>> JB
>>
>> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <dh...@apache.org> wrote:
>> >
>> > Hi All,
>> >
>> > One big emerging enterprise use case coming up as more people
>> consolidate
>> > Data Lakehouses and Catalogs is something commonly known as "Data
>> Sharing",
>> > an more specifically over the course of adoption of open table formats
>> > "Open Data Sharing".
>> >
>> > Examples of existing managed service providers' Data Sharing features:
>> >
>> > https://www.databricks.com/product/delta-sharing
>> > https://docs.snowflake.com/en/user-guide/data-sharing-intro
>> > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
>> > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
>> >
>> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
>> >
>> > The basic idea is that when you share data between different companies,
>> you
>> > need a first-class governance/management layer and extra
>> bells-and-whistles
>> > that are distinct from just the basic capabilities of RBAC or
>> generalized
>> > access-control (i.e. if you're sharing across partially-untrusted org
>> > boundaries, you don't just let the consumer organization log into your
>> > datalake like one of your own employees).
>> >
>> > JB and I put together this high-level proposal for supporting Open
>> Sharing
>> > in Polaris:
>> >
>> >
>> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
>> >
>> > Tentatively, it means adding ~5 logical data model constructs, some of
>> > which may be a first-class PolarisEntity type, others subtypes of
>> existing
>> > entities, and others just a nested construct:
>> >
>> > - ShareEntity (would behave similarly to a Catalog)
>> > - ExternalConsumer (mostly inherits from Principal)
>> > - Listing (Similar to a "role grant" but has different metadata)
>> > - EndpointConfig (nested config under Listing)
>> > - ShareMembership (Similar to a "securable grant" but different
>> metadata)
>> >
>> > Feedback/comments welcome! I'll also bring it up for live discussion if
>> > there's time in the community sync.
>> >
>> > Cheers,
>> > Dennis
>>
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Dennis Huo <dh...@apache.org>.
Thanks JB for reviving this thread, and belated thanks to all reviewers who
have left feedback in the doc or this thread so far! And apologies for the
silence on this, it's still been on my radar but I got distracted by some
other work for awhile.
Echoing JB's summary, the proposal is indeed that we're mostly just reusing
existing Authz/RBAC machinery, and the translation layer to the new sharing
concepts sits *on top of* the existing authz engine as much as possible.
So even if we're adding layers of indirection on ExternalConsumer/Listing
-> Principal/PrincipalRole/CatalogRole+grants, the parts that end up doing
the underlying authz just reuse existing Authorizer logic.
Based on some other offline discussion with Dmitri, I realized why the
discussion around "requirements" may have been confusing:
1. My attempts to shorten the doc inadvertently cut out some of the more
"abstract requirements" formulations
2. The project in itself is somewhat more of a "form *as* function" feature
than *strictly* functional in nature -- for example, many correctly pointed
out that "sharing" via the existing RBAC/Principals syntax is already
*possible* just by letting a cross-org "consumer" be a first-class
principal and access your own company's main Polaris catalog endpoints.
For (1) I added this to the Requirements section in the doc to better
clarify the abstract form of the data model element:
*A. Sharing Container*: Container concept for holding things you want to
share - “ShareEntity”
*B. Consumer Identity*: Special flavor of principal-like entity
representing a consumer relationship - “ExternalConsumer”
*C. Map Containers to Consumers*: A way to specify letting a given consumer
access a given sharing container - “Listing”
*D. Consumer Endpoint*: A segregated API access-point for consumers -
“EndpointConfig”
*E. Objects in Containers*: A way to specify which in-catalog objects
belong in the big sharing container - “ShareMembership”
This still bakes in a bit of implementation detail since it begs the
question why not just let core RBAC/grants handle (C) and (E). To that end,
and addressing problem (2) of where the "form *is* function", I added a
"Concept Mapping" tab:
https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0
I think the details of more advanced features can still be hashed out
incrementally without blocking work on an MVP (and certainly the MVP scope
could be trimmed so we can launch-and-iterate), so as long as there's
community alignment on the general direction I propose we could move
forward on basic building blocks. Or if building blocks themselves are
controversial we could still proceed with some prototyping across different
options.
Cheers,
Dennis
On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi everyone,
>
> I would like to resume the discussion about Data Sharing in Polaris
> and propose focusing on three major items to help us move forward.
>
> 1. What is data sharing?
> In a nutshell, it is an open protocol proposal allowing
> organizations to share live data without copying or replication. The
> main goal is zero-copy sharing achieved through shared metadata and
> authorization.
>
> On Polaris, this would mean:
> a. The consumer is authenticated as usual.
> b. The Polaris server provides the consumer with direct access to the
> data files (e.g., Parquet, as specified in the metadata.json).
> c. The consumer reads the data files directly using their preferred
> tool (e.g., a query engine).
>
> While Polaris already has most of the necessary underlying
> capabilities, we need clear branding/packaging around this use case.
> Specifically, this means:
> a. A share is a dedicated catalog in the Polaris server, or a catalog
> role scoped to specific namespaces.
> b. A recipient is represented by a principal and a principal role.
> c. A consumer profile consists of the catalog URI and the OAuth2
> client_id/client_secret.
> d. We use credential vending (temporary STS credentials) for consumer
> access to the data files.
> e. We use the Polaris IRC layer to expose the metadata and the file
> locations/access.
> f. We can also support Polaris-to-Polaris sharing via Federation.
>
> Concretely:
>
> - On the provider side, we use the standard Polaris RBAC model
> (assigning grants to catalog roles, and assigning those catalog roles
> to principal roles):
> - Create a dedicated principal with its own client_id/client_secret.
> - Create a catalog role with TABLE_READ_DATA, TABLE_LIST, and
> NAMESPACE_LIST grants on the namespaces to be shared (no write
> grants).
> - Create a principal role per destination, attach the catalog
> role, and assign the principal.
> - The destination engine uses loadTable with the
> "X-Iceberg-Access-Delegation: vended-credentials" header. Polaris then
> emits temporary, read-only storage credentials, allowing the engine to
> read the data files directly from the owner's bucket without accessing
> the rest of the bucket. This achieves zero-copy sharing.
> - On the recipient/consumer side, no special configuration is
> needed; any IRC client can use it.
>
> 2. What is data sharing not?
> I previously considered adding an Arrow Flight endpoint to
> Polaris, but I no longer think it is a good idea for the initial
> phase. An Arrow Flight endpoint would serve the data directly, which
> would require Polaris to read the data and expose it. I am against
> embedding a query engine in Polaris, as this tight coupling feels like
> an anti-pattern. While we could use a delegation service to serve the
> Flight endpoint, I think it should be considered for a later phase, if
> at all, since a Flight endpoint might be better suited as a consumer
> responsibility and could encourage data copying.
>
> The first phase should focus on packaging and branding the
> capabilities we already have.
>
> 3. What could the Data Sharing "package" look like in Polaris?
> To make Data Sharing obvious and easy to use, we should introduce
> the missing concept of a "share"—a named definition that users can
> list, discover, and define. The goal is to materialize the user's
> intent ("I want to share these tables with this consumer until this
> date") into standard Polaris entities (principal, principal role,
> catalog role, and grants).
>
> Therefore, I propose adding a /shares endpoint to the Polaris
> Management API to package and materialize these share definitions into
> Polaris entities. This would provide a usable, first-version package
> for Polaris Data Sharing.
>
> I hope this addresses Dmitri's questions and provides a more concrete
> scenario for Polaris Data Sharing.
>
> Thoughts?
>
> Regards,
> JB
>
> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <dh...@apache.org> wrote:
> >
> > Hi All,
> >
> > One big emerging enterprise use case coming up as more people consolidate
> > Data Lakehouses and Catalogs is something commonly known as "Data
> Sharing",
> > an more specifically over the course of adoption of open table formats
> > "Open Data Sharing".
> >
> > Examples of existing managed service providers' Data Sharing features:
> >
> > https://www.databricks.com/product/delta-sharing
> > https://docs.snowflake.com/en/user-guide/data-sharing-intro
> > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> >
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
> >
> > The basic idea is that when you share data between different companies,
> you
> > need a first-class governance/management layer and extra
> bells-and-whistles
> > that are distinct from just the basic capabilities of RBAC or generalized
> > access-control (i.e. if you're sharing across partially-untrusted org
> > boundaries, you don't just let the consumer organization log into your
> > datalake like one of your own employees).
> >
> > JB and I put together this high-level proposal for supporting Open
> Sharing
> > in Polaris:
> >
> >
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
> >
> > Tentatively, it means adding ~5 logical data model constructs, some of
> > which may be a first-class PolarisEntity type, others subtypes of
> existing
> > entities, and others just a nested construct:
> >
> > - ShareEntity (would behave similarly to a Catalog)
> > - ExternalConsumer (mostly inherits from Principal)
> > - Listing (Similar to a "role grant" but has different metadata)
> > - EndpointConfig (nested config under Listing)
> > - ShareMembership (Similar to a "securable grant" but different
> metadata)
> >
> > Feedback/comments welcome! I'll also bring it up for live discussion if
> > there's time in the community sync.
> >
> > Cheers,
> > Dennis
>
Re: [DISCUSS] Adding support for new "Open Sharing" APIs in Polaris
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi everyone,
I would like to resume the discussion about Data Sharing in Polaris
and propose focusing on three major items to help us move forward.
1. What is data sharing?
In a nutshell, it is an open protocol proposal allowing
organizations to share live data without copying or replication. The
main goal is zero-copy sharing achieved through shared metadata and
authorization.
On Polaris, this would mean:
a. The consumer is authenticated as usual.
b. The Polaris server provides the consumer with direct access to the
data files (e.g., Parquet, as specified in the metadata.json).
c. The consumer reads the data files directly using their preferred
tool (e.g., a query engine).
While Polaris already has most of the necessary underlying
capabilities, we need clear branding/packaging around this use case.
Specifically, this means:
a. A share is a dedicated catalog in the Polaris server, or a catalog
role scoped to specific namespaces.
b. A recipient is represented by a principal and a principal role.
c. A consumer profile consists of the catalog URI and the OAuth2
client_id/client_secret.
d. We use credential vending (temporary STS credentials) for consumer
access to the data files.
e. We use the Polaris IRC layer to expose the metadata and the file
locations/access.
f. We can also support Polaris-to-Polaris sharing via Federation.
Concretely:
- On the provider side, we use the standard Polaris RBAC model
(assigning grants to catalog roles, and assigning those catalog roles
to principal roles):
- Create a dedicated principal with its own client_id/client_secret.
- Create a catalog role with TABLE_READ_DATA, TABLE_LIST, and
NAMESPACE_LIST grants on the namespaces to be shared (no write
grants).
- Create a principal role per destination, attach the catalog
role, and assign the principal.
- The destination engine uses loadTable with the
"X-Iceberg-Access-Delegation: vended-credentials" header. Polaris then
emits temporary, read-only storage credentials, allowing the engine to
read the data files directly from the owner's bucket without accessing
the rest of the bucket. This achieves zero-copy sharing.
- On the recipient/consumer side, no special configuration is
needed; any IRC client can use it.
2. What is data sharing not?
I previously considered adding an Arrow Flight endpoint to
Polaris, but I no longer think it is a good idea for the initial
phase. An Arrow Flight endpoint would serve the data directly, which
would require Polaris to read the data and expose it. I am against
embedding a query engine in Polaris, as this tight coupling feels like
an anti-pattern. While we could use a delegation service to serve the
Flight endpoint, I think it should be considered for a later phase, if
at all, since a Flight endpoint might be better suited as a consumer
responsibility and could encourage data copying.
The first phase should focus on packaging and branding the
capabilities we already have.
3. What could the Data Sharing "package" look like in Polaris?
To make Data Sharing obvious and easy to use, we should introduce
the missing concept of a "share"—a named definition that users can
list, discover, and define. The goal is to materialize the user's
intent ("I want to share these tables with this consumer until this
date") into standard Polaris entities (principal, principal role,
catalog role, and grants).
Therefore, I propose adding a /shares endpoint to the Polaris
Management API to package and materialize these share definitions into
Polaris entities. This would provide a usable, first-version package
for Polaris Data Sharing.
I hope this addresses Dmitri's questions and provides a more concrete
scenario for Polaris Data Sharing.
Thoughts?
Regards,
JB
On Thu, May 28, 2026 at 7:46 AM Dennis Huo <dh...@apache.org> wrote:
>
> Hi All,
>
> One big emerging enterprise use case coming up as more people consolidate
> Data Lakehouses and Catalogs is something commonly known as "Data Sharing",
> an more specifically over the course of adoption of open table formats
> "Open Data Sharing".
>
> Examples of existing managed service providers' Data Sharing features:
>
> https://www.databricks.com/product/delta-sharing
> https://docs.snowflake.com/en/user-guide/data-sharing-intro
> https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
>
> The basic idea is that when you share data between different companies, you
> need a first-class governance/management layer and extra bells-and-whistles
> that are distinct from just the basic capabilities of RBAC or generalized
> access-control (i.e. if you're sharing across partially-untrusted org
> boundaries, you don't just let the consumer organization log into your
> datalake like one of your own employees).
>
> JB and I put together this high-level proposal for supporting Open Sharing
> in Polaris:
>
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
>
> Tentatively, it means adding ~5 logical data model constructs, some of
> which may be a first-class PolarisEntity type, others subtypes of existing
> entities, and others just a nested construct:
>
> - ShareEntity (would behave similarly to a Catalog)
> - ExternalConsumer (mostly inherits from Principal)
> - Listing (Similar to a "role grant" but has different metadata)
> - EndpointConfig (nested config under Listing)
> - ShareMembership (Similar to a "securable grant" but different metadata)
>
> Feedback/comments welcome! I'll also bring it up for live discussion if
> there's time in the community sync.
>
> Cheers,
> Dennis