You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Sung Yun <su...@apache.org> on 2026/06/05 20:59:08 UTC

[DISCUSS] Write-path gap for field-id-bound policy during schema evolution

Hi folks,

The FGAC / Read Restriction proposal [1] is introducing a read-side path to
standardize how we describe row filters and masks, and to do it safely
across schema evolution by binding them to field-ids. We don't yet have
anything matching on the write path.

Today, policies are administered entirely outside the REST protocol, so
external systems reference columns by name, as they're not part of the
commit and never see field-ids. And two things break once schema and policy
have to change together:
- a policy bound to a column name silently re-targets when the column is
renamed
- a policy commits separately from the schema change it depends on, so a
column can exist before its protection does

So far, policy administration has been left out of scope [2], and now that
the Read Restrictions Proposal is finding consensus, I believe it is a good
time to start thinking about it on the write path.
I have a rough direction in mind, of enabling co-committing policy and
binding it to field-ids on the server-side. So I wanted to gauge:
1. whether people see this as a gap worth closing in the IRC protocol
2. whether there are concerns or considerations that should be taken into
account

If there's interest, I'm happy to put together a detailed proposal and
share it here for discussion.

Sung

[1]
https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38
[2] https://lists.apache.org/thread/2jx33fn7lq37oxxm7sd6rjy0dnvbm4t6

Re: [DISCUSS] Write-path gap for field-id-bound policy during schema evolution

Posted by Sung Yun <su...@apache.org>.
Thanks to everyone who joined the catalog sync and weighed in. The discussion did not converge, but it clarified the landscape and the next step. Here's a summary of the main points, for those who were not on the call:

- Out-of-band approaches were floated, including client-side pre-assignment of field-ids so that policies can be authored against those ids ahead of the commit. (Ryan, Dan)
- The /transaction endpoint, or something modeled similarly, was raised as a more general mechanism for committing multiple IRC objects together to provide atomicity, with a Policy object as one such object. (Ryan, Dan)
- Several noted that a Policy object would likely need to be a first-class IRC concept before transaction semantics for policy could be discussed. (Russell, Ryan, Yufei, Prashant)
- The difficulty of standardizing the diverse space of existing policy-authoring grammars was raised as an open problem. (Ryan, Prashant)
- A relationship was drawn to the Labels proposal as another way of attaching policy semantics to column ids. (Ryan)

As a next step, I will draft a proposal for a first-class Policy object in the IRC, so the community can assess whether that abstraction can capture the range of policy-authoring patterns in use today.

Thanks everyone again,
Sung

On 2026/06/30 19:54:40 Sung Yun wrote:
> Hi Yufei,
> 
> That's the model I'm going after, where catalogs that want to support it can, as an opt-in rather than a mandate. The IRC has no path today for a catalog to support that option. The commit that introduces the column is sent by the engine, and the catalog cannot attach a policy to a commit it receives rather than authors. So enabling the option is itself a protocol question, and that's what I'm hoping the doc helps us discuss.
> 
> I'll bring this to the catalog community sync tomorrow, and I'm looking forward to picking up the points raised across the thread so far.
> 
> Sung
> 
> On 2026/06/22 17:19:36 Yufei Gu wrote:
> > I tend to agree that this is mostly a catalog implementation concern.
> > 
> > In practice, policy changes are rarely committed in the same transaction as
> > table DDL. That said, a catalog could choose to support that model if it
> > wants stronger guarantees around column creation/update and policy
> > attachment.
> > 
> > So I don’t think the IRC necessarily needs to mandate this behavior. It
> > feels reasonable to leave it as a catalog capability or implementation
> > choice.
> > 
> > Yufei
> > 
> > 
> > On Wed, Jun 17, 2026 at 7:42 AM Sung Yun <su...@apache.org> wrote:
> > 
> > > Hi Prashant, thanks for taking a look at the doc and for sharing the link.
> > >
> > > I agree with you that binding policies using field-ids is the best way to
> > > address the "Drift" problem I've codified in the doc. What I'm hoping to
> > > discuss is that since the field-id doesn't exist until the schema change is
> > > committed, there's a window of time when a column intended to be protected
> > > is not, until that policy is attached out of band.
> > >
> > > The question I hope to answer with the community is whether this is a
> > > limitation we're okay accepting, or whether we want to allow a column to
> > > land already protected, in the same commit that creates it.
> > > I'm hoping the catalog sync will help clarify the problem statement. And
> > > likewise, looking forward to discussing it more.
> > >
> > > Sung
> > >
> > > On Tue, Jun 16, 2026 at 3:39 PM Prashant Singh <pr...@gmail.com>
> > > wrote:
> > >
> > >> Hey Sung,
> > >> I understand people define and attach policies by name but I don't think
> > >> engines / metastore keep names as metadata (at least for the engine /
> > >> catalogs i have worked with). These column names are resolved to field id
> > >> before mapping of policy to table is persisted, this means for
> > >> attachment column must exist.
> > >> Much like one creates an iceberg table with column names and those are
> > >> assigned fieldId which are kind of opaque to the user, then for all
> > >> operations fieldID becomes our source of truth and preserves things like
> > >> rename.
> > >>
> > >> We discussed this exact scenario when we were modeling ReadRestrictions
> > >> as well and made ReadRestrictions return name, and left it to be catalog's
> > >> choice for their metadata representation (09/10/25) :
> > >> https://www.youtube.com/watch?v=orAXA5e9pmU&t=2867s as ideal would be to
> > >> just rely on field id in the first place.
> > >> So it's intentionally left that way since defining policy and attaching
> > >> policy are entirely catalog concerns and how catalog comes with read
> > >> restrictions is entirely catalog implementer choice, I personally don't see
> > >> a gap here.
> > >>
> > >> With that being said this doesn't mean we can't rediscuss, thank you for
> > >> putting to agenda for sync. Looking forward to it.
> > >>
> > >> Best,
> > >> Prashant Singh
> > >>
> > >> On Tue, Jun 16, 2026 at 8:14 AM Sung Yun <su...@apache.org> wrote:
> > >>
> > >>> Hi Andrei, that's a sharp framing, and I think you've identified
> > >>> something broader that spans multiple constructs being discussed today. I
> > >>> agree that it's worth discussing the meta pattern as its own topic.
> > >>>
> > >>> On the data governance side: before we settle what a shared write-side
> > >>> shape should look like, I think it would help to first establish whether
> > >>> the specific problems are ones the community agrees are worth solving. The
> > >>> Drift and Creation problems I raised in the Google Doc have security
> > >>> consequences for FGAC policies, and whether they merit a first-class
> > >>> construct in the IRC write path is a data-governance question that I think
> > >>> is worth putting to the community on its own terms.
> > >>>
> > >>> I'll bring the policy case to the IRC sync, and I'm glad to dig into the
> > >>> meta pattern with you and Sebastian and the rest of the community as it
> > >>> sharpens.
> > >>>
> > >>> Sung
> > >>>
> > >>> On 2026/06/16 12:13:12 Andrei Tserakhau via dev wrote:
> > >>> > Hi Sung,
> > >>> >
> > >>> > Thanks for raising this. The overlap with the labels write-side
> > >>> > work I've been drafting with @Sebastian Baunsgaard
> > >>> > <se...@databricks.com>  is structural --
> > >>> > same lifecycle, field-id binding, co-commit and concurrency questions,
> > >>> > different payload.
> > >>> >
> > >>> > But what stands out more than the specific overlap is that this is
> > >>> > the first sighting of a pattern the spec doesn't yet have a
> > >>> > framework for: catalog-authored, lifecycle-managed write APIs that
> > >>> > reach deep into catalog-owned space. Read Restrictions already
> > >>> > does this on the read side — policies are very much catalog
> > >>> > territory. Your authoring proposal extends it to write. Labels
> > >>> > CRUD lands in the same neighborhood from a different direction.
> > >>> >
> > >>> > Kevin asked something close to this at the May 28 labels sync:
> > >>> > what's the pattern for introducing new first-class concepts in the
> > >>> > REST spec? Ryan's answer pointed at the shape (CRUD verb +
> > >>> > transactional path), but the deeper question hasn't been worked
> > >>> > through — should the spec standardize the write side of
> > >>> > catalog-owned territory at all, or is this best left ad-hoc per
> > >>> > proposal with capability negotiation governing client expectations?
> > >>> >
> > >>> > I lean toward Capabilities being the right frame here. Catalogs
> > >>> > opt in, clients discover what's supported, the spec doesn't force
> > >>> > standardization deep into catalog territory. A unified write-side
> > >>> > surface has real value for clients — engines, custom tools, one
> > >>> > shape to learn — but real cost too: catalog innovation space
> > >>> > shrinks to differentiators inside a spec-prescribed envelope.
> > >>> >
> > >>> > So before aligning on specific conventions, worth asking the
> > >>> > meta-question: shall we go this direction at all? And if yes —
> > >>> > ad-hoc per proposal, or a deliberate meta-framework?
> > >>> >
> > >>> > This is broader than labels alone, so probably worth raising at
> > >>> > one of the upcoming catalog community syncs as a meta-topic
> > >>> > rather than the labels-specific sync. Labels sync can pick up
> > >>> > labels-side implications afterward, once the broader direction
> > >>> > is clearer.
> > >>> >
> > >>> > Best,
> > >>> > Andrei
> > >>> >
> > >>> > On Mon, Jun 15, 2026 at 9:10 AM Sung Yun <su...@apache.org> wrote:
> > >>> >
> > >>> > > Hi Dan,
> > >>> > >
> > >>> > > Apologies for the confusion. "Write" was a poor word choice on my
> > >>> part. I
> > >>> > > didn't mean enforcing policy on writers and you're right that a
> > >>> writer
> > >>> > > holds the highest level of access, and there's little to restrict
> > >>> there. By
> > >>> > > "write" I meant to refer to the lifecycle of the policies themselves:
> > >>> > > creating, updating, and deleting them. Enforcement stays on the read
> > >>> side
> > >>> > > (#13879). This is the complementary authoring path discussion on
> > >>> policies.
> > >>> > >
> > >>> > > The problem I'm looking at is that a policy bound to a column by
> > >>> name can
> > >>> > > detach or retarget when that column is renamed or dropped. A policy
> > >>> also
> > >>> > > can't land in the same commit as the column it protects, so the
> > >>> column can
> > >>> > > exist before its protection does. I've written up the analysis and a
> > >>> > > direction that could close it [1], and I'd appreciate your review.
> > >>> > >
> > >>> > > Christian, thanks. I agree with your pointers. Drop+re-add is a good
> > >>> > > example of the general case and it faces the same exposure as any
> > >>> schema
> > >>> > > change when policy is managed separately from the schema change that
> > >>> > > introduces it, which is exactly the problem the doc works through.
> > >>> I'd
> > >>> > > value your review on the shared doc.
> > >>> > >
> > >>> > > Sung
> > >>> > >
> > >>> > > [1]
> > >>> > >
> > >>> https://docs.google.com/document/d/1yL2Yv70hJ569dpLdW_upTzzK8Zb3fAFEKEH4JRdosjU/edit?tab=t.0
> > >>> > >
> > >>> > > On 2026/06/15 15:19:12 Daniel Weeks wrote:
> > >>> > > > Hey Sung,
> > >>> > > >
> > >>> > > > I'm not sure I fully understand the use case here.  Generally,
> > >>> readers
> > >>> > > can
> > >>> > > > have different policies when they consume data (what's
> > >>> > > > restricted/hidden/obfuscated).  However, on the write path, I'm
> > >>> not aware
> > >>> > > > of scenarios where similar policies would be applied.  A writer
> > >>> typically
> > >>> > > > has the highest level of access because they need to read
> > >>> (metadata at
> > >>> > > > minimum) and write (both metadata and data).
> > >>> > > >
> > >>> > > > What use cases are you envisioning for write side policy
> > >>> enforcement?
> > >>> > > >
> > >>> > > > Thanks,
> > >>> > > > Dan
> > >>> > > >
> > >>> > > > On Sun, Jun 14, 2026 at 11:43 PM Christian Thiel <
> > >>> > > christian.t.b101@gmail.com>
> > >>> > > > wrote:
> > >>> > > >
> > >>> > > > > Hello Sung,
> > >>> > > > >
> > >>> > > > > thanks for sharing this!
> > >>> > > > >
> > >>> > > > > I'd definitely be interested in seeing your ideas for the
> > >>> proposal.
> > >>> > > > > Especially your point about field-id binding had me thinking —
> > >>> since
> > >>> > > admins
> > >>> > > > > author against names and never see field-ids today, it'd be worth
> > >>> > > spelling
> > >>> > > > > out where and when that name→field-id binding happens, and how it
> > >>> > > handles
> > >>> > > > > drop+re-add.
> > >>> > > > >
> > >>> > > > > I think a number of interesting points are worth discussing such
> > >>> as
> > >>> > > > > coexistence with external policy engines and separation of
> > >>> duties on
> > >>> > > > > commit, while still keeping the field-id binding intact where it
> > >>> > > applies.
> > >>> > > > >
> > >>> > > > > Looking forward to it!
> > >>> > > > >
> > >>> > > > > Best,
> > >>> > > > > Christian
> > >>> > > > >
> > >>> > > > > On Fri, 5 Jun 2026 at 22:59, Sung Yun <su...@apache.org> wrote:
> > >>> > > > >
> > >>> > > > >> Hi folks,
> > >>> > > > >>
> > >>> > > > >> The FGAC / Read Restriction proposal [1] is introducing a
> > >>> read-side
> > >>> > > path
> > >>> > > > >> to standardize how we describe row filters and masks, and to do
> > >>> it
> > >>> > > safely
> > >>> > > > >> across schema evolution by binding them to field-ids. We don't
> > >>> yet
> > >>> > > have
> > >>> > > > >> anything matching on the write path.
> > >>> > > > >>
> > >>> > > > >> Today, policies are administered entirely outside the REST
> > >>> protocol,
> > >>> > > so
> > >>> > > > >> external systems reference columns by name, as they're not part
> > >>> of the
> > >>> > > > >> commit and never see field-ids. And two things break once
> > >>> schema and
> > >>> > > policy
> > >>> > > > >> have to change together:
> > >>> > > > >> - a policy bound to a column name silently re-targets when the
> > >>> column
> > >>> > > is
> > >>> > > > >> renamed
> > >>> > > > >> - a policy commits separately from the schema change it depends
> > >>> on,
> > >>> > > so a
> > >>> > > > >> column can exist before its protection does
> > >>> > > > >>
> > >>> > > > >> So far, policy administration has been left out of scope [2],
> > >>> and now
> > >>> > > > >> that the Read Restrictions Proposal is finding consensus, I
> > >>> believe
> > >>> > > it is a
> > >>> > > > >> good time to start thinking about it on the write path.
> > >>> > > > >> I have a rough direction in mind, of enabling co-committing
> > >>> policy and
> > >>> > > > >> binding it to field-ids on the server-side. So I wanted to
> > >>> gauge:
> > >>> > > > >> 1. whether people see this as a gap worth closing in the IRC
> > >>> protocol
> > >>> > > > >> 2. whether there are concerns or considerations that should be
> > >>> taken
> > >>> > > into
> > >>> > > > >> account
> > >>> > > > >>
> > >>> > > > >> If there's interest, I'm happy to put together a detailed
> > >>> proposal and
> > >>> > > > >> share it here for discussion.
> > >>> > > > >>
> > >>> > > > >> Sung
> > >>> > > > >>
> > >>> > > > >> [1]
> > >>> > > > >>
> > >>> > >
> > >>> https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38
> > >>> > > > >> [2]
> > >>> https://lists.apache.org/thread/2jx33fn7lq37oxxm7sd6rjy0dnvbm4t6
> > >>> > > > >>
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> > 
> 

Re: [DISCUSS] Write-path gap for field-id-bound policy during schema evolution

Posted by Sung Yun <su...@apache.org>.
Hi Yufei,

That's the model I'm going after, where catalogs that want to support it can, as an opt-in rather than a mandate. The IRC has no path today for a catalog to support that option. The commit that introduces the column is sent by the engine, and the catalog cannot attach a policy to a commit it receives rather than authors. So enabling the option is itself a protocol question, and that's what I'm hoping the doc helps us discuss.

I'll bring this to the catalog community sync tomorrow, and I'm looking forward to picking up the points raised across the thread so far.

Sung

On 2026/06/22 17:19:36 Yufei Gu wrote:
> I tend to agree that this is mostly a catalog implementation concern.
> 
> In practice, policy changes are rarely committed in the same transaction as
> table DDL. That said, a catalog could choose to support that model if it
> wants stronger guarantees around column creation/update and policy
> attachment.
> 
> So I don’t think the IRC necessarily needs to mandate this behavior. It
> feels reasonable to leave it as a catalog capability or implementation
> choice.
> 
> Yufei
> 
> 
> On Wed, Jun 17, 2026 at 7:42 AM Sung Yun <su...@apache.org> wrote:
> 
> > Hi Prashant, thanks for taking a look at the doc and for sharing the link.
> >
> > I agree with you that binding policies using field-ids is the best way to
> > address the "Drift" problem I've codified in the doc. What I'm hoping to
> > discuss is that since the field-id doesn't exist until the schema change is
> > committed, there's a window of time when a column intended to be protected
> > is not, until that policy is attached out of band.
> >
> > The question I hope to answer with the community is whether this is a
> > limitation we're okay accepting, or whether we want to allow a column to
> > land already protected, in the same commit that creates it.
> > I'm hoping the catalog sync will help clarify the problem statement. And
> > likewise, looking forward to discussing it more.
> >
> > Sung
> >
> > On Tue, Jun 16, 2026 at 3:39 PM Prashant Singh <pr...@gmail.com>
> > wrote:
> >
> >> Hey Sung,
> >> I understand people define and attach policies by name but I don't think
> >> engines / metastore keep names as metadata (at least for the engine /
> >> catalogs i have worked with). These column names are resolved to field id
> >> before mapping of policy to table is persisted, this means for
> >> attachment column must exist.
> >> Much like one creates an iceberg table with column names and those are
> >> assigned fieldId which are kind of opaque to the user, then for all
> >> operations fieldID becomes our source of truth and preserves things like
> >> rename.
> >>
> >> We discussed this exact scenario when we were modeling ReadRestrictions
> >> as well and made ReadRestrictions return name, and left it to be catalog's
> >> choice for their metadata representation (09/10/25) :
> >> https://www.youtube.com/watch?v=orAXA5e9pmU&t=2867s as ideal would be to
> >> just rely on field id in the first place.
> >> So it's intentionally left that way since defining policy and attaching
> >> policy are entirely catalog concerns and how catalog comes with read
> >> restrictions is entirely catalog implementer choice, I personally don't see
> >> a gap here.
> >>
> >> With that being said this doesn't mean we can't rediscuss, thank you for
> >> putting to agenda for sync. Looking forward to it.
> >>
> >> Best,
> >> Prashant Singh
> >>
> >> On Tue, Jun 16, 2026 at 8:14 AM Sung Yun <su...@apache.org> wrote:
> >>
> >>> Hi Andrei, that's a sharp framing, and I think you've identified
> >>> something broader that spans multiple constructs being discussed today. I
> >>> agree that it's worth discussing the meta pattern as its own topic.
> >>>
> >>> On the data governance side: before we settle what a shared write-side
> >>> shape should look like, I think it would help to first establish whether
> >>> the specific problems are ones the community agrees are worth solving. The
> >>> Drift and Creation problems I raised in the Google Doc have security
> >>> consequences for FGAC policies, and whether they merit a first-class
> >>> construct in the IRC write path is a data-governance question that I think
> >>> is worth putting to the community on its own terms.
> >>>
> >>> I'll bring the policy case to the IRC sync, and I'm glad to dig into the
> >>> meta pattern with you and Sebastian and the rest of the community as it
> >>> sharpens.
> >>>
> >>> Sung
> >>>
> >>> On 2026/06/16 12:13:12 Andrei Tserakhau via dev wrote:
> >>> > Hi Sung,
> >>> >
> >>> > Thanks for raising this. The overlap with the labels write-side
> >>> > work I've been drafting with @Sebastian Baunsgaard
> >>> > <se...@databricks.com>  is structural --
> >>> > same lifecycle, field-id binding, co-commit and concurrency questions,
> >>> > different payload.
> >>> >
> >>> > But what stands out more than the specific overlap is that this is
> >>> > the first sighting of a pattern the spec doesn't yet have a
> >>> > framework for: catalog-authored, lifecycle-managed write APIs that
> >>> > reach deep into catalog-owned space. Read Restrictions already
> >>> > does this on the read side — policies are very much catalog
> >>> > territory. Your authoring proposal extends it to write. Labels
> >>> > CRUD lands in the same neighborhood from a different direction.
> >>> >
> >>> > Kevin asked something close to this at the May 28 labels sync:
> >>> > what's the pattern for introducing new first-class concepts in the
> >>> > REST spec? Ryan's answer pointed at the shape (CRUD verb +
> >>> > transactional path), but the deeper question hasn't been worked
> >>> > through — should the spec standardize the write side of
> >>> > catalog-owned territory at all, or is this best left ad-hoc per
> >>> > proposal with capability negotiation governing client expectations?
> >>> >
> >>> > I lean toward Capabilities being the right frame here. Catalogs
> >>> > opt in, clients discover what's supported, the spec doesn't force
> >>> > standardization deep into catalog territory. A unified write-side
> >>> > surface has real value for clients — engines, custom tools, one
> >>> > shape to learn — but real cost too: catalog innovation space
> >>> > shrinks to differentiators inside a spec-prescribed envelope.
> >>> >
> >>> > So before aligning on specific conventions, worth asking the
> >>> > meta-question: shall we go this direction at all? And if yes —
> >>> > ad-hoc per proposal, or a deliberate meta-framework?
> >>> >
> >>> > This is broader than labels alone, so probably worth raising at
> >>> > one of the upcoming catalog community syncs as a meta-topic
> >>> > rather than the labels-specific sync. Labels sync can pick up
> >>> > labels-side implications afterward, once the broader direction
> >>> > is clearer.
> >>> >
> >>> > Best,
> >>> > Andrei
> >>> >
> >>> > On Mon, Jun 15, 2026 at 9:10 AM Sung Yun <su...@apache.org> wrote:
> >>> >
> >>> > > Hi Dan,
> >>> > >
> >>> > > Apologies for the confusion. "Write" was a poor word choice on my
> >>> part. I
> >>> > > didn't mean enforcing policy on writers and you're right that a
> >>> writer
> >>> > > holds the highest level of access, and there's little to restrict
> >>> there. By
> >>> > > "write" I meant to refer to the lifecycle of the policies themselves:
> >>> > > creating, updating, and deleting them. Enforcement stays on the read
> >>> side
> >>> > > (#13879). This is the complementary authoring path discussion on
> >>> policies.
> >>> > >
> >>> > > The problem I'm looking at is that a policy bound to a column by
> >>> name can
> >>> > > detach or retarget when that column is renamed or dropped. A policy
> >>> also
> >>> > > can't land in the same commit as the column it protects, so the
> >>> column can
> >>> > > exist before its protection does. I've written up the analysis and a
> >>> > > direction that could close it [1], and I'd appreciate your review.
> >>> > >
> >>> > > Christian, thanks. I agree with your pointers. Drop+re-add is a good
> >>> > > example of the general case and it faces the same exposure as any
> >>> schema
> >>> > > change when policy is managed separately from the schema change that
> >>> > > introduces it, which is exactly the problem the doc works through.
> >>> I'd
> >>> > > value your review on the shared doc.
> >>> > >
> >>> > > Sung
> >>> > >
> >>> > > [1]
> >>> > >
> >>> https://docs.google.com/document/d/1yL2Yv70hJ569dpLdW_upTzzK8Zb3fAFEKEH4JRdosjU/edit?tab=t.0
> >>> > >
> >>> > > On 2026/06/15 15:19:12 Daniel Weeks wrote:
> >>> > > > Hey Sung,
> >>> > > >
> >>> > > > I'm not sure I fully understand the use case here.  Generally,
> >>> readers
> >>> > > can
> >>> > > > have different policies when they consume data (what's
> >>> > > > restricted/hidden/obfuscated).  However, on the write path, I'm
> >>> not aware
> >>> > > > of scenarios where similar policies would be applied.  A writer
> >>> typically
> >>> > > > has the highest level of access because they need to read
> >>> (metadata at
> >>> > > > minimum) and write (both metadata and data).
> >>> > > >
> >>> > > > What use cases are you envisioning for write side policy
> >>> enforcement?
> >>> > > >
> >>> > > > Thanks,
> >>> > > > Dan
> >>> > > >
> >>> > > > On Sun, Jun 14, 2026 at 11:43 PM Christian Thiel <
> >>> > > christian.t.b101@gmail.com>
> >>> > > > wrote:
> >>> > > >
> >>> > > > > Hello Sung,
> >>> > > > >
> >>> > > > > thanks for sharing this!
> >>> > > > >
> >>> > > > > I'd definitely be interested in seeing your ideas for the
> >>> proposal.
> >>> > > > > Especially your point about field-id binding had me thinking —
> >>> since
> >>> > > admins
> >>> > > > > author against names and never see field-ids today, it'd be worth
> >>> > > spelling
> >>> > > > > out where and when that name→field-id binding happens, and how it
> >>> > > handles
> >>> > > > > drop+re-add.
> >>> > > > >
> >>> > > > > I think a number of interesting points are worth discussing such
> >>> as
> >>> > > > > coexistence with external policy engines and separation of
> >>> duties on
> >>> > > > > commit, while still keeping the field-id binding intact where it
> >>> > > applies.
> >>> > > > >
> >>> > > > > Looking forward to it!
> >>> > > > >
> >>> > > > > Best,
> >>> > > > > Christian
> >>> > > > >
> >>> > > > > On Fri, 5 Jun 2026 at 22:59, Sung Yun <su...@apache.org> wrote:
> >>> > > > >
> >>> > > > >> Hi folks,
> >>> > > > >>
> >>> > > > >> The FGAC / Read Restriction proposal [1] is introducing a
> >>> read-side
> >>> > > path
> >>> > > > >> to standardize how we describe row filters and masks, and to do
> >>> it
> >>> > > safely
> >>> > > > >> across schema evolution by binding them to field-ids. We don't
> >>> yet
> >>> > > have
> >>> > > > >> anything matching on the write path.
> >>> > > > >>
> >>> > > > >> Today, policies are administered entirely outside the REST
> >>> protocol,
> >>> > > so
> >>> > > > >> external systems reference columns by name, as they're not part
> >>> of the
> >>> > > > >> commit and never see field-ids. And two things break once
> >>> schema and
> >>> > > policy
> >>> > > > >> have to change together:
> >>> > > > >> - a policy bound to a column name silently re-targets when the
> >>> column
> >>> > > is
> >>> > > > >> renamed
> >>> > > > >> - a policy commits separately from the schema change it depends
> >>> on,
> >>> > > so a
> >>> > > > >> column can exist before its protection does
> >>> > > > >>
> >>> > > > >> So far, policy administration has been left out of scope [2],
> >>> and now
> >>> > > > >> that the Read Restrictions Proposal is finding consensus, I
> >>> believe
> >>> > > it is a
> >>> > > > >> good time to start thinking about it on the write path.
> >>> > > > >> I have a rough direction in mind, of enabling co-committing
> >>> policy and
> >>> > > > >> binding it to field-ids on the server-side. So I wanted to
> >>> gauge:
> >>> > > > >> 1. whether people see this as a gap worth closing in the IRC
> >>> protocol
> >>> > > > >> 2. whether there are concerns or considerations that should be
> >>> taken
> >>> > > into
> >>> > > > >> account
> >>> > > > >>
> >>> > > > >> If there's interest, I'm happy to put together a detailed
> >>> proposal and
> >>> > > > >> share it here for discussion.
> >>> > > > >>
> >>> > > > >> Sung
> >>> > > > >>
> >>> > > > >> [1]
> >>> > > > >>
> >>> > >
> >>> https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38
> >>> > > > >> [2]
> >>> https://lists.apache.org/thread/2jx33fn7lq37oxxm7sd6rjy0dnvbm4t6
> >>> > > > >>
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> 

Re: [DISCUSS] Write-path gap for field-id-bound policy during schema evolution

Posted by Yufei Gu <fl...@gmail.com>.
I tend to agree that this is mostly a catalog implementation concern.

In practice, policy changes are rarely committed in the same transaction as
table DDL. That said, a catalog could choose to support that model if it
wants stronger guarantees around column creation/update and policy
attachment.

So I don’t think the IRC necessarily needs to mandate this behavior. It
feels reasonable to leave it as a catalog capability or implementation
choice.

Yufei


On Wed, Jun 17, 2026 at 7:42 AM Sung Yun <su...@apache.org> wrote:

> Hi Prashant, thanks for taking a look at the doc and for sharing the link.
>
> I agree with you that binding policies using field-ids is the best way to
> address the "Drift" problem I've codified in the doc. What I'm hoping to
> discuss is that since the field-id doesn't exist until the schema change is
> committed, there's a window of time when a column intended to be protected
> is not, until that policy is attached out of band.
>
> The question I hope to answer with the community is whether this is a
> limitation we're okay accepting, or whether we want to allow a column to
> land already protected, in the same commit that creates it.
> I'm hoping the catalog sync will help clarify the problem statement. And
> likewise, looking forward to discussing it more.
>
> Sung
>
> On Tue, Jun 16, 2026 at 3:39 PM Prashant Singh <pr...@gmail.com>
> wrote:
>
>> Hey Sung,
>> I understand people define and attach policies by name but I don't think
>> engines / metastore keep names as metadata (at least for the engine /
>> catalogs i have worked with). These column names are resolved to field id
>> before mapping of policy to table is persisted, this means for
>> attachment column must exist.
>> Much like one creates an iceberg table with column names and those are
>> assigned fieldId which are kind of opaque to the user, then for all
>> operations fieldID becomes our source of truth and preserves things like
>> rename.
>>
>> We discussed this exact scenario when we were modeling ReadRestrictions
>> as well and made ReadRestrictions return name, and left it to be catalog's
>> choice for their metadata representation (09/10/25) :
>> https://www.youtube.com/watch?v=orAXA5e9pmU&t=2867s as ideal would be to
>> just rely on field id in the first place.
>> So it's intentionally left that way since defining policy and attaching
>> policy are entirely catalog concerns and how catalog comes with read
>> restrictions is entirely catalog implementer choice, I personally don't see
>> a gap here.
>>
>> With that being said this doesn't mean we can't rediscuss, thank you for
>> putting to agenda for sync. Looking forward to it.
>>
>> Best,
>> Prashant Singh
>>
>> On Tue, Jun 16, 2026 at 8:14 AM Sung Yun <su...@apache.org> wrote:
>>
>>> Hi Andrei, that's a sharp framing, and I think you've identified
>>> something broader that spans multiple constructs being discussed today. I
>>> agree that it's worth discussing the meta pattern as its own topic.
>>>
>>> On the data governance side: before we settle what a shared write-side
>>> shape should look like, I think it would help to first establish whether
>>> the specific problems are ones the community agrees are worth solving. The
>>> Drift and Creation problems I raised in the Google Doc have security
>>> consequences for FGAC policies, and whether they merit a first-class
>>> construct in the IRC write path is a data-governance question that I think
>>> is worth putting to the community on its own terms.
>>>
>>> I'll bring the policy case to the IRC sync, and I'm glad to dig into the
>>> meta pattern with you and Sebastian and the rest of the community as it
>>> sharpens.
>>>
>>> Sung
>>>
>>> On 2026/06/16 12:13:12 Andrei Tserakhau via dev wrote:
>>> > Hi Sung,
>>> >
>>> > Thanks for raising this. The overlap with the labels write-side
>>> > work I've been drafting with @Sebastian Baunsgaard
>>> > <se...@databricks.com>  is structural --
>>> > same lifecycle, field-id binding, co-commit and concurrency questions,
>>> > different payload.
>>> >
>>> > But what stands out more than the specific overlap is that this is
>>> > the first sighting of a pattern the spec doesn't yet have a
>>> > framework for: catalog-authored, lifecycle-managed write APIs that
>>> > reach deep into catalog-owned space. Read Restrictions already
>>> > does this on the read side — policies are very much catalog
>>> > territory. Your authoring proposal extends it to write. Labels
>>> > CRUD lands in the same neighborhood from a different direction.
>>> >
>>> > Kevin asked something close to this at the May 28 labels sync:
>>> > what's the pattern for introducing new first-class concepts in the
>>> > REST spec? Ryan's answer pointed at the shape (CRUD verb +
>>> > transactional path), but the deeper question hasn't been worked
>>> > through — should the spec standardize the write side of
>>> > catalog-owned territory at all, or is this best left ad-hoc per
>>> > proposal with capability negotiation governing client expectations?
>>> >
>>> > I lean toward Capabilities being the right frame here. Catalogs
>>> > opt in, clients discover what's supported, the spec doesn't force
>>> > standardization deep into catalog territory. A unified write-side
>>> > surface has real value for clients — engines, custom tools, one
>>> > shape to learn — but real cost too: catalog innovation space
>>> > shrinks to differentiators inside a spec-prescribed envelope.
>>> >
>>> > So before aligning on specific conventions, worth asking the
>>> > meta-question: shall we go this direction at all? And if yes —
>>> > ad-hoc per proposal, or a deliberate meta-framework?
>>> >
>>> > This is broader than labels alone, so probably worth raising at
>>> > one of the upcoming catalog community syncs as a meta-topic
>>> > rather than the labels-specific sync. Labels sync can pick up
>>> > labels-side implications afterward, once the broader direction
>>> > is clearer.
>>> >
>>> > Best,
>>> > Andrei
>>> >
>>> > On Mon, Jun 15, 2026 at 9:10 AM Sung Yun <su...@apache.org> wrote:
>>> >
>>> > > Hi Dan,
>>> > >
>>> > > Apologies for the confusion. "Write" was a poor word choice on my
>>> part. I
>>> > > didn't mean enforcing policy on writers and you're right that a
>>> writer
>>> > > holds the highest level of access, and there's little to restrict
>>> there. By
>>> > > "write" I meant to refer to the lifecycle of the policies themselves:
>>> > > creating, updating, and deleting them. Enforcement stays on the read
>>> side
>>> > > (#13879). This is the complementary authoring path discussion on
>>> policies.
>>> > >
>>> > > The problem I'm looking at is that a policy bound to a column by
>>> name can
>>> > > detach or retarget when that column is renamed or dropped. A policy
>>> also
>>> > > can't land in the same commit as the column it protects, so the
>>> column can
>>> > > exist before its protection does. I've written up the analysis and a
>>> > > direction that could close it [1], and I'd appreciate your review.
>>> > >
>>> > > Christian, thanks. I agree with your pointers. Drop+re-add is a good
>>> > > example of the general case and it faces the same exposure as any
>>> schema
>>> > > change when policy is managed separately from the schema change that
>>> > > introduces it, which is exactly the problem the doc works through.
>>> I'd
>>> > > value your review on the shared doc.
>>> > >
>>> > > Sung
>>> > >
>>> > > [1]
>>> > >
>>> https://docs.google.com/document/d/1yL2Yv70hJ569dpLdW_upTzzK8Zb3fAFEKEH4JRdosjU/edit?tab=t.0
>>> > >
>>> > > On 2026/06/15 15:19:12 Daniel Weeks wrote:
>>> > > > Hey Sung,
>>> > > >
>>> > > > I'm not sure I fully understand the use case here.  Generally,
>>> readers
>>> > > can
>>> > > > have different policies when they consume data (what's
>>> > > > restricted/hidden/obfuscated).  However, on the write path, I'm
>>> not aware
>>> > > > of scenarios where similar policies would be applied.  A writer
>>> typically
>>> > > > has the highest level of access because they need to read
>>> (metadata at
>>> > > > minimum) and write (both metadata and data).
>>> > > >
>>> > > > What use cases are you envisioning for write side policy
>>> enforcement?
>>> > > >
>>> > > > Thanks,
>>> > > > Dan
>>> > > >
>>> > > > On Sun, Jun 14, 2026 at 11:43 PM Christian Thiel <
>>> > > christian.t.b101@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > Hello Sung,
>>> > > > >
>>> > > > > thanks for sharing this!
>>> > > > >
>>> > > > > I'd definitely be interested in seeing your ideas for the
>>> proposal.
>>> > > > > Especially your point about field-id binding had me thinking —
>>> since
>>> > > admins
>>> > > > > author against names and never see field-ids today, it'd be worth
>>> > > spelling
>>> > > > > out where and when that name→field-id binding happens, and how it
>>> > > handles
>>> > > > > drop+re-add.
>>> > > > >
>>> > > > > I think a number of interesting points are worth discussing such
>>> as
>>> > > > > coexistence with external policy engines and separation of
>>> duties on
>>> > > > > commit, while still keeping the field-id binding intact where it
>>> > > applies.
>>> > > > >
>>> > > > > Looking forward to it!
>>> > > > >
>>> > > > > Best,
>>> > > > > Christian
>>> > > > >
>>> > > > > On Fri, 5 Jun 2026 at 22:59, Sung Yun <su...@apache.org> wrote:
>>> > > > >
>>> > > > >> Hi folks,
>>> > > > >>
>>> > > > >> The FGAC / Read Restriction proposal [1] is introducing a
>>> read-side
>>> > > path
>>> > > > >> to standardize how we describe row filters and masks, and to do
>>> it
>>> > > safely
>>> > > > >> across schema evolution by binding them to field-ids. We don't
>>> yet
>>> > > have
>>> > > > >> anything matching on the write path.
>>> > > > >>
>>> > > > >> Today, policies are administered entirely outside the REST
>>> protocol,
>>> > > so
>>> > > > >> external systems reference columns by name, as they're not part
>>> of the
>>> > > > >> commit and never see field-ids. And two things break once
>>> schema and
>>> > > policy
>>> > > > >> have to change together:
>>> > > > >> - a policy bound to a column name silently re-targets when the
>>> column
>>> > > is
>>> > > > >> renamed
>>> > > > >> - a policy commits separately from the schema change it depends
>>> on,
>>> > > so a
>>> > > > >> column can exist before its protection does
>>> > > > >>
>>> > > > >> So far, policy administration has been left out of scope [2],
>>> and now
>>> > > > >> that the Read Restrictions Proposal is finding consensus, I
>>> believe
>>> > > it is a
>>> > > > >> good time to start thinking about it on the write path.
>>> > > > >> I have a rough direction in mind, of enabling co-committing
>>> policy and
>>> > > > >> binding it to field-ids on the server-side. So I wanted to
>>> gauge:
>>> > > > >> 1. whether people see this as a gap worth closing in the IRC
>>> protocol
>>> > > > >> 2. whether there are concerns or considerations that should be
>>> taken
>>> > > into
>>> > > > >> account
>>> > > > >>
>>> > > > >> If there's interest, I'm happy to put together a detailed
>>> proposal and
>>> > > > >> share it here for discussion.
>>> > > > >>
>>> > > > >> Sung
>>> > > > >>
>>> > > > >> [1]
>>> > > > >>
>>> > >
>>> https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38
>>> > > > >> [2]
>>> https://lists.apache.org/thread/2jx33fn7lq37oxxm7sd6rjy0dnvbm4t6
>>> > > > >>
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>

Re: [DISCUSS] Write-path gap for field-id-bound policy during schema evolution

Posted by Sung Yun <su...@apache.org>.
Hi Prashant, thanks for taking a look at the doc and for sharing the link.

I agree with you that binding policies using field-ids is the best way to
address the "Drift" problem I've codified in the doc. What I'm hoping to
discuss is that since the field-id doesn't exist until the schema change is
committed, there's a window of time when a column intended to be protected
is not, until that policy is attached out of band.

The question I hope to answer with the community is whether this is a
limitation we're okay accepting, or whether we want to allow a column to
land already protected, in the same commit that creates it.
I'm hoping the catalog sync will help clarify the problem statement. And
likewise, looking forward to discussing it more.

Sung

On Tue, Jun 16, 2026 at 3:39 PM Prashant Singh <pr...@gmail.com>
wrote:

> Hey Sung,
> I understand people define and attach policies by name but I don't think
> engines / metastore keep names as metadata (at least for the engine /
> catalogs i have worked with). These column names are resolved to field id
> before mapping of policy to table is persisted, this means for
> attachment column must exist.
> Much like one creates an iceberg table with column names and those are
> assigned fieldId which are kind of opaque to the user, then for all
> operations fieldID becomes our source of truth and preserves things like
> rename.
>
> We discussed this exact scenario when we were modeling ReadRestrictions as
> well and made ReadRestrictions return name, and left it to be catalog's
> choice for their metadata representation (09/10/25) :
> https://www.youtube.com/watch?v=orAXA5e9pmU&t=2867s as ideal would be to
> just rely on field id in the first place.
> So it's intentionally left that way since defining policy and attaching
> policy are entirely catalog concerns and how catalog comes with read
> restrictions is entirely catalog implementer choice, I personally don't see
> a gap here.
>
> With that being said this doesn't mean we can't rediscuss, thank you for
> putting to agenda for sync. Looking forward to it.
>
> Best,
> Prashant Singh
>
> On Tue, Jun 16, 2026 at 8:14 AM Sung Yun <su...@apache.org> wrote:
>
>> Hi Andrei, that's a sharp framing, and I think you've identified
>> something broader that spans multiple constructs being discussed today. I
>> agree that it's worth discussing the meta pattern as its own topic.
>>
>> On the data governance side: before we settle what a shared write-side
>> shape should look like, I think it would help to first establish whether
>> the specific problems are ones the community agrees are worth solving. The
>> Drift and Creation problems I raised in the Google Doc have security
>> consequences for FGAC policies, and whether they merit a first-class
>> construct in the IRC write path is a data-governance question that I think
>> is worth putting to the community on its own terms.
>>
>> I'll bring the policy case to the IRC sync, and I'm glad to dig into the
>> meta pattern with you and Sebastian and the rest of the community as it
>> sharpens.
>>
>> Sung
>>
>> On 2026/06/16 12:13:12 Andrei Tserakhau via dev wrote:
>> > Hi Sung,
>> >
>> > Thanks for raising this. The overlap with the labels write-side
>> > work I've been drafting with @Sebastian Baunsgaard
>> > <se...@databricks.com>  is structural --
>> > same lifecycle, field-id binding, co-commit and concurrency questions,
>> > different payload.
>> >
>> > But what stands out more than the specific overlap is that this is
>> > the first sighting of a pattern the spec doesn't yet have a
>> > framework for: catalog-authored, lifecycle-managed write APIs that
>> > reach deep into catalog-owned space. Read Restrictions already
>> > does this on the read side — policies are very much catalog
>> > territory. Your authoring proposal extends it to write. Labels
>> > CRUD lands in the same neighborhood from a different direction.
>> >
>> > Kevin asked something close to this at the May 28 labels sync:
>> > what's the pattern for introducing new first-class concepts in the
>> > REST spec? Ryan's answer pointed at the shape (CRUD verb +
>> > transactional path), but the deeper question hasn't been worked
>> > through — should the spec standardize the write side of
>> > catalog-owned territory at all, or is this best left ad-hoc per
>> > proposal with capability negotiation governing client expectations?
>> >
>> > I lean toward Capabilities being the right frame here. Catalogs
>> > opt in, clients discover what's supported, the spec doesn't force
>> > standardization deep into catalog territory. A unified write-side
>> > surface has real value for clients — engines, custom tools, one
>> > shape to learn — but real cost too: catalog innovation space
>> > shrinks to differentiators inside a spec-prescribed envelope.
>> >
>> > So before aligning on specific conventions, worth asking the
>> > meta-question: shall we go this direction at all? And if yes —
>> > ad-hoc per proposal, or a deliberate meta-framework?
>> >
>> > This is broader than labels alone, so probably worth raising at
>> > one of the upcoming catalog community syncs as a meta-topic
>> > rather than the labels-specific sync. Labels sync can pick up
>> > labels-side implications afterward, once the broader direction
>> > is clearer.
>> >
>> > Best,
>> > Andrei
>> >
>> > On Mon, Jun 15, 2026 at 9:10 AM Sung Yun <su...@apache.org> wrote:
>> >
>> > > Hi Dan,
>> > >
>> > > Apologies for the confusion. "Write" was a poor word choice on my
>> part. I
>> > > didn't mean enforcing policy on writers and you're right that a writer
>> > > holds the highest level of access, and there's little to restrict
>> there. By
>> > > "write" I meant to refer to the lifecycle of the policies themselves:
>> > > creating, updating, and deleting them. Enforcement stays on the read
>> side
>> > > (#13879). This is the complementary authoring path discussion on
>> policies.
>> > >
>> > > The problem I'm looking at is that a policy bound to a column by name
>> can
>> > > detach or retarget when that column is renamed or dropped. A policy
>> also
>> > > can't land in the same commit as the column it protects, so the
>> column can
>> > > exist before its protection does. I've written up the analysis and a
>> > > direction that could close it [1], and I'd appreciate your review.
>> > >
>> > > Christian, thanks. I agree with your pointers. Drop+re-add is a good
>> > > example of the general case and it faces the same exposure as any
>> schema
>> > > change when policy is managed separately from the schema change that
>> > > introduces it, which is exactly the problem the doc works through. I'd
>> > > value your review on the shared doc.
>> > >
>> > > Sung
>> > >
>> > > [1]
>> > >
>> https://docs.google.com/document/d/1yL2Yv70hJ569dpLdW_upTzzK8Zb3fAFEKEH4JRdosjU/edit?tab=t.0
>> > >
>> > > On 2026/06/15 15:19:12 Daniel Weeks wrote:
>> > > > Hey Sung,
>> > > >
>> > > > I'm not sure I fully understand the use case here.  Generally,
>> readers
>> > > can
>> > > > have different policies when they consume data (what's
>> > > > restricted/hidden/obfuscated).  However, on the write path, I'm not
>> aware
>> > > > of scenarios where similar policies would be applied.  A writer
>> typically
>> > > > has the highest level of access because they need to read (metadata
>> at
>> > > > minimum) and write (both metadata and data).
>> > > >
>> > > > What use cases are you envisioning for write side policy
>> enforcement?
>> > > >
>> > > > Thanks,
>> > > > Dan
>> > > >
>> > > > On Sun, Jun 14, 2026 at 11:43 PM Christian Thiel <
>> > > christian.t.b101@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hello Sung,
>> > > > >
>> > > > > thanks for sharing this!
>> > > > >
>> > > > > I'd definitely be interested in seeing your ideas for the
>> proposal.
>> > > > > Especially your point about field-id binding had me thinking —
>> since
>> > > admins
>> > > > > author against names and never see field-ids today, it'd be worth
>> > > spelling
>> > > > > out where and when that name→field-id binding happens, and how it
>> > > handles
>> > > > > drop+re-add.
>> > > > >
>> > > > > I think a number of interesting points are worth discussing such
>> as
>> > > > > coexistence with external policy engines and separation of duties
>> on
>> > > > > commit, while still keeping the field-id binding intact where it
>> > > applies.
>> > > > >
>> > > > > Looking forward to it!
>> > > > >
>> > > > > Best,
>> > > > > Christian
>> > > > >
>> > > > > On Fri, 5 Jun 2026 at 22:59, Sung Yun <su...@apache.org> wrote:
>> > > > >
>> > > > >> Hi folks,
>> > > > >>
>> > > > >> The FGAC / Read Restriction proposal [1] is introducing a
>> read-side
>> > > path
>> > > > >> to standardize how we describe row filters and masks, and to do
>> it
>> > > safely
>> > > > >> across schema evolution by binding them to field-ids. We don't
>> yet
>> > > have
>> > > > >> anything matching on the write path.
>> > > > >>
>> > > > >> Today, policies are administered entirely outside the REST
>> protocol,
>> > > so
>> > > > >> external systems reference columns by name, as they're not part
>> of the
>> > > > >> commit and never see field-ids. And two things break once schema
>> and
>> > > policy
>> > > > >> have to change together:
>> > > > >> - a policy bound to a column name silently re-targets when the
>> column
>> > > is
>> > > > >> renamed
>> > > > >> - a policy commits separately from the schema change it depends
>> on,
>> > > so a
>> > > > >> column can exist before its protection does
>> > > > >>
>> > > > >> So far, policy administration has been left out of scope [2],
>> and now
>> > > > >> that the Read Restrictions Proposal is finding consensus, I
>> believe
>> > > it is a
>> > > > >> good time to start thinking about it on the write path.
>> > > > >> I have a rough direction in mind, of enabling co-committing
>> policy and
>> > > > >> binding it to field-ids on the server-side. So I wanted to gauge:
>> > > > >> 1. whether people see this as a gap worth closing in the IRC
>> protocol
>> > > > >> 2. whether there are concerns or considerations that should be
>> taken
>> > > into
>> > > > >> account
>> > > > >>
>> > > > >> If there's interest, I'm happy to put together a detailed
>> proposal and
>> > > > >> share it here for discussion.
>> > > > >>
>> > > > >> Sung
>> > > > >>
>> > > > >> [1]
>> > > > >>
>> > >
>> https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38
>> > > > >> [2]
>> https://lists.apache.org/thread/2jx33fn7lq37oxxm7sd6rjy0dnvbm4t6
>> > > > >>
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] Write-path gap for field-id-bound policy during schema evolution

Posted by Prashant Singh <pr...@gmail.com>.
Hey Sung,
I understand people define and attach policies by name but I don't think
engines / metastore keep names as metadata (at least for the engine /
catalogs i have worked with). These column names are resolved to field id
before mapping of policy to table is persisted, this means for
attachment column must exist.
Much like one creates an iceberg table with column names and those are
assigned fieldId which are kind of opaque to the user, then for all
operations fieldID becomes our source of truth and preserves things like
rename.

We discussed this exact scenario when we were modeling ReadRestrictions as
well and made ReadRestrictions return name, and left it to be catalog's
choice for their metadata representation (09/10/25) :
https://www.youtube.com/watch?v=orAXA5e9pmU&t=2867s as ideal would be to
just rely on field id in the first place.
So it's intentionally left that way since defining policy and attaching
policy are entirely catalog concerns and how catalog comes with read
restrictions is entirely catalog implementer choice, I personally don't see
a gap here.

With that being said this doesn't mean we can't rediscuss, thank you for
putting to agenda for sync. Looking forward to it.

Best,
Prashant Singh

On Tue, Jun 16, 2026 at 8:14 AM Sung Yun <su...@apache.org> wrote:

> Hi Andrei, that's a sharp framing, and I think you've identified something
> broader that spans multiple constructs being discussed today. I agree that
> it's worth discussing the meta pattern as its own topic.
>
> On the data governance side: before we settle what a shared write-side
> shape should look like, I think it would help to first establish whether
> the specific problems are ones the community agrees are worth solving. The
> Drift and Creation problems I raised in the Google Doc have security
> consequences for FGAC policies, and whether they merit a first-class
> construct in the IRC write path is a data-governance question that I think
> is worth putting to the community on its own terms.
>
> I'll bring the policy case to the IRC sync, and I'm glad to dig into the
> meta pattern with you and Sebastian and the rest of the community as it
> sharpens.
>
> Sung
>
> On 2026/06/16 12:13:12 Andrei Tserakhau via dev wrote:
> > Hi Sung,
> >
> > Thanks for raising this. The overlap with the labels write-side
> > work I've been drafting with @Sebastian Baunsgaard
> > <se...@databricks.com>  is structural --
> > same lifecycle, field-id binding, co-commit and concurrency questions,
> > different payload.
> >
> > But what stands out more than the specific overlap is that this is
> > the first sighting of a pattern the spec doesn't yet have a
> > framework for: catalog-authored, lifecycle-managed write APIs that
> > reach deep into catalog-owned space. Read Restrictions already
> > does this on the read side — policies are very much catalog
> > territory. Your authoring proposal extends it to write. Labels
> > CRUD lands in the same neighborhood from a different direction.
> >
> > Kevin asked something close to this at the May 28 labels sync:
> > what's the pattern for introducing new first-class concepts in the
> > REST spec? Ryan's answer pointed at the shape (CRUD verb +
> > transactional path), but the deeper question hasn't been worked
> > through — should the spec standardize the write side of
> > catalog-owned territory at all, or is this best left ad-hoc per
> > proposal with capability negotiation governing client expectations?
> >
> > I lean toward Capabilities being the right frame here. Catalogs
> > opt in, clients discover what's supported, the spec doesn't force
> > standardization deep into catalog territory. A unified write-side
> > surface has real value for clients — engines, custom tools, one
> > shape to learn — but real cost too: catalog innovation space
> > shrinks to differentiators inside a spec-prescribed envelope.
> >
> > So before aligning on specific conventions, worth asking the
> > meta-question: shall we go this direction at all? And if yes —
> > ad-hoc per proposal, or a deliberate meta-framework?
> >
> > This is broader than labels alone, so probably worth raising at
> > one of the upcoming catalog community syncs as a meta-topic
> > rather than the labels-specific sync. Labels sync can pick up
> > labels-side implications afterward, once the broader direction
> > is clearer.
> >
> > Best,
> > Andrei
> >
> > On Mon, Jun 15, 2026 at 9:10 AM Sung Yun <su...@apache.org> wrote:
> >
> > > Hi Dan,
> > >
> > > Apologies for the confusion. "Write" was a poor word choice on my
> part. I
> > > didn't mean enforcing policy on writers and you're right that a writer
> > > holds the highest level of access, and there's little to restrict
> there. By
> > > "write" I meant to refer to the lifecycle of the policies themselves:
> > > creating, updating, and deleting them. Enforcement stays on the read
> side
> > > (#13879). This is the complementary authoring path discussion on
> policies.
> > >
> > > The problem I'm looking at is that a policy bound to a column by name
> can
> > > detach or retarget when that column is renamed or dropped. A policy
> also
> > > can't land in the same commit as the column it protects, so the column
> can
> > > exist before its protection does. I've written up the analysis and a
> > > direction that could close it [1], and I'd appreciate your review.
> > >
> > > Christian, thanks. I agree with your pointers. Drop+re-add is a good
> > > example of the general case and it faces the same exposure as any
> schema
> > > change when policy is managed separately from the schema change that
> > > introduces it, which is exactly the problem the doc works through. I'd
> > > value your review on the shared doc.
> > >
> > > Sung
> > >
> > > [1]
> > >
> https://docs.google.com/document/d/1yL2Yv70hJ569dpLdW_upTzzK8Zb3fAFEKEH4JRdosjU/edit?tab=t.0
> > >
> > > On 2026/06/15 15:19:12 Daniel Weeks wrote:
> > > > Hey Sung,
> > > >
> > > > I'm not sure I fully understand the use case here.  Generally,
> readers
> > > can
> > > > have different policies when they consume data (what's
> > > > restricted/hidden/obfuscated).  However, on the write path, I'm not
> aware
> > > > of scenarios where similar policies would be applied.  A writer
> typically
> > > > has the highest level of access because they need to read (metadata
> at
> > > > minimum) and write (both metadata and data).
> > > >
> > > > What use cases are you envisioning for write side policy enforcement?
> > > >
> > > > Thanks,
> > > > Dan
> > > >
> > > > On Sun, Jun 14, 2026 at 11:43 PM Christian Thiel <
> > > christian.t.b101@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello Sung,
> > > > >
> > > > > thanks for sharing this!
> > > > >
> > > > > I'd definitely be interested in seeing your ideas for the proposal.
> > > > > Especially your point about field-id binding had me thinking —
> since
> > > admins
> > > > > author against names and never see field-ids today, it'd be worth
> > > spelling
> > > > > out where and when that name→field-id binding happens, and how it
> > > handles
> > > > > drop+re-add.
> > > > >
> > > > > I think a number of interesting points are worth discussing such as
> > > > > coexistence with external policy engines and separation of duties
> on
> > > > > commit, while still keeping the field-id binding intact where it
> > > applies.
> > > > >
> > > > > Looking forward to it!
> > > > >
> > > > > Best,
> > > > > Christian
> > > > >
> > > > > On Fri, 5 Jun 2026 at 22:59, Sung Yun <su...@apache.org> wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> The FGAC / Read Restriction proposal [1] is introducing a
> read-side
> > > path
> > > > >> to standardize how we describe row filters and masks, and to do it
> > > safely
> > > > >> across schema evolution by binding them to field-ids. We don't yet
> > > have
> > > > >> anything matching on the write path.
> > > > >>
> > > > >> Today, policies are administered entirely outside the REST
> protocol,
> > > so
> > > > >> external systems reference columns by name, as they're not part
> of the
> > > > >> commit and never see field-ids. And two things break once schema
> and
> > > policy
> > > > >> have to change together:
> > > > >> - a policy bound to a column name silently re-targets when the
> column
> > > is
> > > > >> renamed
> > > > >> - a policy commits separately from the schema change it depends
> on,
> > > so a
> > > > >> column can exist before its protection does
> > > > >>
> > > > >> So far, policy administration has been left out of scope [2], and
> now
> > > > >> that the Read Restrictions Proposal is finding consensus, I
> believe
> > > it is a
> > > > >> good time to start thinking about it on the write path.
> > > > >> I have a rough direction in mind, of enabling co-committing
> policy and
> > > > >> binding it to field-ids on the server-side. So I wanted to gauge:
> > > > >> 1. whether people see this as a gap worth closing in the IRC
> protocol
> > > > >> 2. whether there are concerns or considerations that should be
> taken
> > > into
> > > > >> account
> > > > >>
> > > > >> If there's interest, I'm happy to put together a detailed
> proposal and
> > > > >> share it here for discussion.
> > > > >>
> > > > >> Sung
> > > > >>
> > > > >> [1]
> > > > >>
> > >
> https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38
> > > > >> [2]
> https://lists.apache.org/thread/2jx33fn7lq37oxxm7sd6rjy0dnvbm4t6
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Write-path gap for field-id-bound policy during schema evolution

Posted by Sung Yun <su...@apache.org>.
Hi Andrei, that's a sharp framing, and I think you've identified something broader that spans multiple constructs being discussed today. I agree that it's worth discussing the meta pattern as its own topic.

On the data governance side: before we settle what a shared write-side shape should look like, I think it would help to first establish whether the specific problems are ones the community agrees are worth solving. The Drift and Creation problems I raised in the Google Doc have security consequences for FGAC policies, and whether they merit a first-class construct in the IRC write path is a data-governance question that I think is worth putting to the community on its own terms.

I'll bring the policy case to the IRC sync, and I'm glad to dig into the meta pattern with you and Sebastian and the rest of the community as it sharpens.

Sung

On 2026/06/16 12:13:12 Andrei Tserakhau via dev wrote:
> Hi Sung,
> 
> Thanks for raising this. The overlap with the labels write-side
> work I've been drafting with @Sebastian Baunsgaard
> <se...@databricks.com>  is structural --
> same lifecycle, field-id binding, co-commit and concurrency questions,
> different payload.
> 
> But what stands out more than the specific overlap is that this is
> the first sighting of a pattern the spec doesn't yet have a
> framework for: catalog-authored, lifecycle-managed write APIs that
> reach deep into catalog-owned space. Read Restrictions already
> does this on the read side — policies are very much catalog
> territory. Your authoring proposal extends it to write. Labels
> CRUD lands in the same neighborhood from a different direction.
> 
> Kevin asked something close to this at the May 28 labels sync:
> what's the pattern for introducing new first-class concepts in the
> REST spec? Ryan's answer pointed at the shape (CRUD verb +
> transactional path), but the deeper question hasn't been worked
> through — should the spec standardize the write side of
> catalog-owned territory at all, or is this best left ad-hoc per
> proposal with capability negotiation governing client expectations?
> 
> I lean toward Capabilities being the right frame here. Catalogs
> opt in, clients discover what's supported, the spec doesn't force
> standardization deep into catalog territory. A unified write-side
> surface has real value for clients — engines, custom tools, one
> shape to learn — but real cost too: catalog innovation space
> shrinks to differentiators inside a spec-prescribed envelope.
> 
> So before aligning on specific conventions, worth asking the
> meta-question: shall we go this direction at all? And if yes —
> ad-hoc per proposal, or a deliberate meta-framework?
> 
> This is broader than labels alone, so probably worth raising at
> one of the upcoming catalog community syncs as a meta-topic
> rather than the labels-specific sync. Labels sync can pick up
> labels-side implications afterward, once the broader direction
> is clearer.
> 
> Best,
> Andrei
> 
> On Mon, Jun 15, 2026 at 9:10 AM Sung Yun <su...@apache.org> wrote:
> 
> > Hi Dan,
> >
> > Apologies for the confusion. "Write" was a poor word choice on my part. I
> > didn't mean enforcing policy on writers and you're right that a writer
> > holds the highest level of access, and there's little to restrict there. By
> > "write" I meant to refer to the lifecycle of the policies themselves:
> > creating, updating, and deleting them. Enforcement stays on the read side
> > (#13879). This is the complementary authoring path discussion on policies.
> >
> > The problem I'm looking at is that a policy bound to a column by name can
> > detach or retarget when that column is renamed or dropped. A policy also
> > can't land in the same commit as the column it protects, so the column can
> > exist before its protection does. I've written up the analysis and a
> > direction that could close it [1], and I'd appreciate your review.
> >
> > Christian, thanks. I agree with your pointers. Drop+re-add is a good
> > example of the general case and it faces the same exposure as any schema
> > change when policy is managed separately from the schema change that
> > introduces it, which is exactly the problem the doc works through. I'd
> > value your review on the shared doc.
> >
> > Sung
> >
> > [1]
> > https://docs.google.com/document/d/1yL2Yv70hJ569dpLdW_upTzzK8Zb3fAFEKEH4JRdosjU/edit?tab=t.0
> >
> > On 2026/06/15 15:19:12 Daniel Weeks wrote:
> > > Hey Sung,
> > >
> > > I'm not sure I fully understand the use case here.  Generally, readers
> > can
> > > have different policies when they consume data (what's
> > > restricted/hidden/obfuscated).  However, on the write path, I'm not aware
> > > of scenarios where similar policies would be applied.  A writer typically
> > > has the highest level of access because they need to read (metadata at
> > > minimum) and write (both metadata and data).
> > >
> > > What use cases are you envisioning for write side policy enforcement?
> > >
> > > Thanks,
> > > Dan
> > >
> > > On Sun, Jun 14, 2026 at 11:43 PM Christian Thiel <
> > christian.t.b101@gmail.com>
> > > wrote:
> > >
> > > > Hello Sung,
> > > >
> > > > thanks for sharing this!
> > > >
> > > > I'd definitely be interested in seeing your ideas for the proposal.
> > > > Especially your point about field-id binding had me thinking — since
> > admins
> > > > author against names and never see field-ids today, it'd be worth
> > spelling
> > > > out where and when that name→field-id binding happens, and how it
> > handles
> > > > drop+re-add.
> > > >
> > > > I think a number of interesting points are worth discussing such as
> > > > coexistence with external policy engines and separation of duties on
> > > > commit, while still keeping the field-id binding intact where it
> > applies.
> > > >
> > > > Looking forward to it!
> > > >
> > > > Best,
> > > > Christian
> > > >
> > > > On Fri, 5 Jun 2026 at 22:59, Sung Yun <su...@apache.org> wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> The FGAC / Read Restriction proposal [1] is introducing a read-side
> > path
> > > >> to standardize how we describe row filters and masks, and to do it
> > safely
> > > >> across schema evolution by binding them to field-ids. We don't yet
> > have
> > > >> anything matching on the write path.
> > > >>
> > > >> Today, policies are administered entirely outside the REST protocol,
> > so
> > > >> external systems reference columns by name, as they're not part of the
> > > >> commit and never see field-ids. And two things break once schema and
> > policy
> > > >> have to change together:
> > > >> - a policy bound to a column name silently re-targets when the column
> > is
> > > >> renamed
> > > >> - a policy commits separately from the schema change it depends on,
> > so a
> > > >> column can exist before its protection does
> > > >>
> > > >> So far, policy administration has been left out of scope [2], and now
> > > >> that the Read Restrictions Proposal is finding consensus, I believe
> > it is a
> > > >> good time to start thinking about it on the write path.
> > > >> I have a rough direction in mind, of enabling co-committing policy and
> > > >> binding it to field-ids on the server-side. So I wanted to gauge:
> > > >> 1. whether people see this as a gap worth closing in the IRC protocol
> > > >> 2. whether there are concerns or considerations that should be taken
> > into
> > > >> account
> > > >>
> > > >> If there's interest, I'm happy to put together a detailed proposal and
> > > >> share it here for discussion.
> > > >>
> > > >> Sung
> > > >>
> > > >> [1]
> > > >>
> > https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38
> > > >> [2] https://lists.apache.org/thread/2jx33fn7lq37oxxm7sd6rjy0dnvbm4t6
> > > >>
> > > >
> > >
> >
> 

Re: [DISCUSS] Write-path gap for field-id-bound policy during schema evolution

Posted by Andrei Tserakhau via dev <de...@iceberg.apache.org>.
Hi Sung,

Thanks for raising this. The overlap with the labels write-side
work I've been drafting with @Sebastian Baunsgaard
<se...@databricks.com>  is structural --
same lifecycle, field-id binding, co-commit and concurrency questions,
different payload.

But what stands out more than the specific overlap is that this is
the first sighting of a pattern the spec doesn't yet have a
framework for: catalog-authored, lifecycle-managed write APIs that
reach deep into catalog-owned space. Read Restrictions already
does this on the read side — policies are very much catalog
territory. Your authoring proposal extends it to write. Labels
CRUD lands in the same neighborhood from a different direction.

Kevin asked something close to this at the May 28 labels sync:
what's the pattern for introducing new first-class concepts in the
REST spec? Ryan's answer pointed at the shape (CRUD verb +
transactional path), but the deeper question hasn't been worked
through — should the spec standardize the write side of
catalog-owned territory at all, or is this best left ad-hoc per
proposal with capability negotiation governing client expectations?

I lean toward Capabilities being the right frame here. Catalogs
opt in, clients discover what's supported, the spec doesn't force
standardization deep into catalog territory. A unified write-side
surface has real value for clients — engines, custom tools, one
shape to learn — but real cost too: catalog innovation space
shrinks to differentiators inside a spec-prescribed envelope.

So before aligning on specific conventions, worth asking the
meta-question: shall we go this direction at all? And if yes —
ad-hoc per proposal, or a deliberate meta-framework?

This is broader than labels alone, so probably worth raising at
one of the upcoming catalog community syncs as a meta-topic
rather than the labels-specific sync. Labels sync can pick up
labels-side implications afterward, once the broader direction
is clearer.

Best,
Andrei

On Mon, Jun 15, 2026 at 9:10 AM Sung Yun <su...@apache.org> wrote:

> Hi Dan,
>
> Apologies for the confusion. "Write" was a poor word choice on my part. I
> didn't mean enforcing policy on writers and you're right that a writer
> holds the highest level of access, and there's little to restrict there. By
> "write" I meant to refer to the lifecycle of the policies themselves:
> creating, updating, and deleting them. Enforcement stays on the read side
> (#13879). This is the complementary authoring path discussion on policies.
>
> The problem I'm looking at is that a policy bound to a column by name can
> detach or retarget when that column is renamed or dropped. A policy also
> can't land in the same commit as the column it protects, so the column can
> exist before its protection does. I've written up the analysis and a
> direction that could close it [1], and I'd appreciate your review.
>
> Christian, thanks. I agree with your pointers. Drop+re-add is a good
> example of the general case and it faces the same exposure as any schema
> change when policy is managed separately from the schema change that
> introduces it, which is exactly the problem the doc works through. I'd
> value your review on the shared doc.
>
> Sung
>
> [1]
> https://docs.google.com/document/d/1yL2Yv70hJ569dpLdW_upTzzK8Zb3fAFEKEH4JRdosjU/edit?tab=t.0
>
> On 2026/06/15 15:19:12 Daniel Weeks wrote:
> > Hey Sung,
> >
> > I'm not sure I fully understand the use case here.  Generally, readers
> can
> > have different policies when they consume data (what's
> > restricted/hidden/obfuscated).  However, on the write path, I'm not aware
> > of scenarios where similar policies would be applied.  A writer typically
> > has the highest level of access because they need to read (metadata at
> > minimum) and write (both metadata and data).
> >
> > What use cases are you envisioning for write side policy enforcement?
> >
> > Thanks,
> > Dan
> >
> > On Sun, Jun 14, 2026 at 11:43 PM Christian Thiel <
> christian.t.b101@gmail.com>
> > wrote:
> >
> > > Hello Sung,
> > >
> > > thanks for sharing this!
> > >
> > > I'd definitely be interested in seeing your ideas for the proposal.
> > > Especially your point about field-id binding had me thinking — since
> admins
> > > author against names and never see field-ids today, it'd be worth
> spelling
> > > out where and when that name→field-id binding happens, and how it
> handles
> > > drop+re-add.
> > >
> > > I think a number of interesting points are worth discussing such as
> > > coexistence with external policy engines and separation of duties on
> > > commit, while still keeping the field-id binding intact where it
> applies.
> > >
> > > Looking forward to it!
> > >
> > > Best,
> > > Christian
> > >
> > > On Fri, 5 Jun 2026 at 22:59, Sung Yun <su...@apache.org> wrote:
> > >
> > >> Hi folks,
> > >>
> > >> The FGAC / Read Restriction proposal [1] is introducing a read-side
> path
> > >> to standardize how we describe row filters and masks, and to do it
> safely
> > >> across schema evolution by binding them to field-ids. We don't yet
> have
> > >> anything matching on the write path.
> > >>
> > >> Today, policies are administered entirely outside the REST protocol,
> so
> > >> external systems reference columns by name, as they're not part of the
> > >> commit and never see field-ids. And two things break once schema and
> policy
> > >> have to change together:
> > >> - a policy bound to a column name silently re-targets when the column
> is
> > >> renamed
> > >> - a policy commits separately from the schema change it depends on,
> so a
> > >> column can exist before its protection does
> > >>
> > >> So far, policy administration has been left out of scope [2], and now
> > >> that the Read Restrictions Proposal is finding consensus, I believe
> it is a
> > >> good time to start thinking about it on the write path.
> > >> I have a rough direction in mind, of enabling co-committing policy and
> > >> binding it to field-ids on the server-side. So I wanted to gauge:
> > >> 1. whether people see this as a gap worth closing in the IRC protocol
> > >> 2. whether there are concerns or considerations that should be taken
> into
> > >> account
> > >>
> > >> If there's interest, I'm happy to put together a detailed proposal and
> > >> share it here for discussion.
> > >>
> > >> Sung
> > >>
> > >> [1]
> > >>
> https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38
> > >> [2] https://lists.apache.org/thread/2jx33fn7lq37oxxm7sd6rjy0dnvbm4t6
> > >>
> > >
> >
>

Re: [DISCUSS] Write-path gap for field-id-bound policy during schema evolution

Posted by Sung Yun <su...@apache.org>.
Hi Dan,

Apologies for the confusion. "Write" was a poor word choice on my part. I didn't mean enforcing policy on writers and you're right that a writer holds the highest level of access, and there's little to restrict there. By "write" I meant to refer to the lifecycle of the policies themselves: creating, updating, and deleting them. Enforcement stays on the read side (#13879). This is the complementary authoring path discussion on policies.

The problem I'm looking at is that a policy bound to a column by name can detach or retarget when that column is renamed or dropped. A policy also can't land in the same commit as the column it protects, so the column can exist before its protection does. I've written up the analysis and a direction that could close it [1], and I'd appreciate your review.

Christian, thanks. I agree with your pointers. Drop+re-add is a good example of the general case and it faces the same exposure as any schema change when policy is managed separately from the schema change that introduces it, which is exactly the problem the doc works through. I'd value your review on the shared doc.

Sung

[1] https://docs.google.com/document/d/1yL2Yv70hJ569dpLdW_upTzzK8Zb3fAFEKEH4JRdosjU/edit?tab=t.0

On 2026/06/15 15:19:12 Daniel Weeks wrote:
> Hey Sung,
> 
> I'm not sure I fully understand the use case here.  Generally, readers can
> have different policies when they consume data (what's
> restricted/hidden/obfuscated).  However, on the write path, I'm not aware
> of scenarios where similar policies would be applied.  A writer typically
> has the highest level of access because they need to read (metadata at
> minimum) and write (both metadata and data).
> 
> What use cases are you envisioning for write side policy enforcement?
> 
> Thanks,
> Dan
> 
> On Sun, Jun 14, 2026 at 11:43 PM Christian Thiel <ch...@gmail.com>
> wrote:
> 
> > Hello Sung,
> >
> > thanks for sharing this!
> >
> > I'd definitely be interested in seeing your ideas for the proposal.
> > Especially your point about field-id binding had me thinking — since admins
> > author against names and never see field-ids today, it'd be worth spelling
> > out where and when that name→field-id binding happens, and how it handles
> > drop+re-add.
> >
> > I think a number of interesting points are worth discussing such as
> > coexistence with external policy engines and separation of duties on
> > commit, while still keeping the field-id binding intact where it applies.
> >
> > Looking forward to it!
> >
> > Best,
> > Christian
> >
> > On Fri, 5 Jun 2026 at 22:59, Sung Yun <su...@apache.org> wrote:
> >
> >> Hi folks,
> >>
> >> The FGAC / Read Restriction proposal [1] is introducing a read-side path
> >> to standardize how we describe row filters and masks, and to do it safely
> >> across schema evolution by binding them to field-ids. We don't yet have
> >> anything matching on the write path.
> >>
> >> Today, policies are administered entirely outside the REST protocol, so
> >> external systems reference columns by name, as they're not part of the
> >> commit and never see field-ids. And two things break once schema and policy
> >> have to change together:
> >> - a policy bound to a column name silently re-targets when the column is
> >> renamed
> >> - a policy commits separately from the schema change it depends on, so a
> >> column can exist before its protection does
> >>
> >> So far, policy administration has been left out of scope [2], and now
> >> that the Read Restrictions Proposal is finding consensus, I believe it is a
> >> good time to start thinking about it on the write path.
> >> I have a rough direction in mind, of enabling co-committing policy and
> >> binding it to field-ids on the server-side. So I wanted to gauge:
> >> 1. whether people see this as a gap worth closing in the IRC protocol
> >> 2. whether there are concerns or considerations that should be taken into
> >> account
> >>
> >> If there's interest, I'm happy to put together a detailed proposal and
> >> share it here for discussion.
> >>
> >> Sung
> >>
> >> [1]
> >> https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38
> >> [2] https://lists.apache.org/thread/2jx33fn7lq37oxxm7sd6rjy0dnvbm4t6
> >>
> >
> 

Re: [DISCUSS] Write-path gap for field-id-bound policy during schema evolution

Posted by Daniel Weeks <dw...@apache.org>.
Hey Sung,

I'm not sure I fully understand the use case here.  Generally, readers can
have different policies when they consume data (what's
restricted/hidden/obfuscated).  However, on the write path, I'm not aware
of scenarios where similar policies would be applied.  A writer typically
has the highest level of access because they need to read (metadata at
minimum) and write (both metadata and data).

What use cases are you envisioning for write side policy enforcement?

Thanks,
Dan

On Sun, Jun 14, 2026 at 11:43 PM Christian Thiel <ch...@gmail.com>
wrote:

> Hello Sung,
>
> thanks for sharing this!
>
> I'd definitely be interested in seeing your ideas for the proposal.
> Especially your point about field-id binding had me thinking — since admins
> author against names and never see field-ids today, it'd be worth spelling
> out where and when that name→field-id binding happens, and how it handles
> drop+re-add.
>
> I think a number of interesting points are worth discussing such as
> coexistence with external policy engines and separation of duties on
> commit, while still keeping the field-id binding intact where it applies.
>
> Looking forward to it!
>
> Best,
> Christian
>
> On Fri, 5 Jun 2026 at 22:59, Sung Yun <su...@apache.org> wrote:
>
>> Hi folks,
>>
>> The FGAC / Read Restriction proposal [1] is introducing a read-side path
>> to standardize how we describe row filters and masks, and to do it safely
>> across schema evolution by binding them to field-ids. We don't yet have
>> anything matching on the write path.
>>
>> Today, policies are administered entirely outside the REST protocol, so
>> external systems reference columns by name, as they're not part of the
>> commit and never see field-ids. And two things break once schema and policy
>> have to change together:
>> - a policy bound to a column name silently re-targets when the column is
>> renamed
>> - a policy commits separately from the schema change it depends on, so a
>> column can exist before its protection does
>>
>> So far, policy administration has been left out of scope [2], and now
>> that the Read Restrictions Proposal is finding consensus, I believe it is a
>> good time to start thinking about it on the write path.
>> I have a rough direction in mind, of enabling co-committing policy and
>> binding it to field-ids on the server-side. So I wanted to gauge:
>> 1. whether people see this as a gap worth closing in the IRC protocol
>> 2. whether there are concerns or considerations that should be taken into
>> account
>>
>> If there's interest, I'm happy to put together a detailed proposal and
>> share it here for discussion.
>>
>> Sung
>>
>> [1]
>> https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38
>> [2] https://lists.apache.org/thread/2jx33fn7lq37oxxm7sd6rjy0dnvbm4t6
>>
>

Re: [DISCUSS] Write-path gap for field-id-bound policy during schema evolution

Posted by Christian Thiel <ch...@gmail.com>.
Hello Sung,

thanks for sharing this!

I'd definitely be interested in seeing your ideas for the proposal.
Especially your point about field-id binding had me thinking — since admins
author against names and never see field-ids today, it'd be worth spelling
out where and when that name→field-id binding happens, and how it handles
drop+re-add.

I think a number of interesting points are worth discussing such as
coexistence with external policy engines and separation of duties on
commit, while still keeping the field-id binding intact where it applies.

Looking forward to it!

Best,
Christian

On Fri, 5 Jun 2026 at 22:59, Sung Yun <su...@apache.org> wrote:

> Hi folks,
>
> The FGAC / Read Restriction proposal [1] is introducing a read-side path
> to standardize how we describe row filters and masks, and to do it safely
> across schema evolution by binding them to field-ids. We don't yet have
> anything matching on the write path.
>
> Today, policies are administered entirely outside the REST protocol, so
> external systems reference columns by name, as they're not part of the
> commit and never see field-ids. And two things break once schema and policy
> have to change together:
> - a policy bound to a column name silently re-targets when the column is
> renamed
> - a policy commits separately from the schema change it depends on, so a
> column can exist before its protection does
>
> So far, policy administration has been left out of scope [2], and now that
> the Read Restrictions Proposal is finding consensus, I believe it is a good
> time to start thinking about it on the write path.
> I have a rough direction in mind, of enabling co-committing policy and
> binding it to field-ids on the server-side. So I wanted to gauge:
> 1. whether people see this as a gap worth closing in the IRC protocol
> 2. whether there are concerns or considerations that should be taken into
> account
>
> If there's interest, I'm happy to put together a detailed proposal and
> share it here for discussion.
>
> Sung
>
> [1]
> https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38
> [2] https://lists.apache.org/thread/2jx33fn7lq37oxxm7sd6rjy0dnvbm4t6
>