[DISCUSS] Catalog-level branches for atomic multi-table promotion (#3469)

[Ankur Kotwal <ko...@gmail.com>]

Hi all,

I'd like to start a discussion on catalog-level branching, tracked in
https://www.google.com/url?q=https://github.com/apache/polaris/issues/3469&source=gmail&ust=1791440827911000&sa=E.
Several users have asked for it
there, and Dmitri suggested taking it to the list. Below is a first
proposal. I'd
much rather shape it with you than arrive with a finished design, so please
poke
holes in it.

1. The problem

Iceberg already lets you branch a single table: a branch is just another
snapshot
ref in the table metadata. That works well for one table, but teams that run
backfills, migrations or "write-audit-publish" pipelines across many tables
hit
three walls:

- Creating or merging a branch across N tables takes N separate calls.
- Promotion isn't atomic. While tables are merged one by one, readers can
see
some tables with the new data and others without it. With hundreds of
tables,
that window is long, and a failure halfway through leaves the catalog in a
mixed
state someone has to repair by hand.
- There's no cheap way to ask "which tables have branch X?" without loading
every
table's metadata.

Today the answer is either to adopt Nessie, or to script it and accept the
non-atomic window.

2. What I'm proposing

A catalog branch: a named, Polaris-managed group of per-table Iceberg
branches
that is created and promoted as a unit.

polaris branch create backfill-2026 --from main --namespace sales
... jobs write to the "backfill-2026" branch of each table as usual ...
polaris branch promote backfill-2026 --into main # all tables, or none

Design goals:

- Engines don't change. Writers and readers use the standard Iceberg branch
API
(for example sales.orders.branch_backfill-2026 in Spark, or the "ref" field
of REST commits). Nothing new is needed in Spark, Flink, Trino or PyIceberg.
- Build on what Polaris already has. Per-table state stays in Iceberg
metadata,
and promotion reuses the transaction workspace that commitTransaction
already
uses to apply changes to several tables as one atomic metastore update.
- Keep the first version small and safe. Fast-forward promotion only,
explicit
failure on conflict, no merge of diverged histories.

3. How it would work

3.1 A new entity

A CATALOG_BRANCH entity, scoped to a catalog, holding:

- the branch name,
- its scope (a namespace, optionally recursive, or an explicit list of
tables),
- for each member table: the table id and the snapshot id of main when the
branch was created (the base),
- created by / created at, and a state (ACTIVE, PROMOTED, DROPPED).

Keeping membership in the entity answers "which tables have branch X?" with
one
read, instead of loading every table's metadata.

3.2 Create

For every table in scope, add an Iceberg branch ref pointing at that table's
current main snapshot, and write the CATALOG_BRANCH entity. All of it goes
through the transaction workspace, so either every table gets the ref and
the
entity exists, or nothing changes.

3.3 Write

Nothing new. Jobs commit to the table's branch ref through the normal REST
updateTable path. Polaris only needs to reject commits that would move a
catalog branch's ref outside of Polaris' control in ways that break
promotion
(for example deleting the ref). Whether that check is worth having in v1 is
one
of the questions below.

3.4 Promote

For every member table, in one transaction:

1. load the current metadata,
2. check that main still points at the recorded base snapshot (fast-forward
check),
3. set main to the branch head,
4. optionally remove the branch ref.

If any table fails the check, nothing is promoted, and the response lists
the
tables that moved since the branch was created. Readers either see the old
state
of every table or the new state of every table.

3.5 Drop

Remove the branch refs from member tables and mark the entity DROPPED.
Snapshot
expiry then cleans up the unreferenced data as it does today.

3.6 API and authorization

A Polaris REST extension under the catalog API (create, list, get, promote,
drop),
plus CLI commands. Two options for authorization: new privileges
(CATALOG_BRANCH_CREATE, CATALOG_BRANCH_PROMOTE), or requiring
TABLE_WRITE_DATA
on every member table. The second is simpler, but slow to check for very
large
branches.

4. Edge cases I've thought about so far

- Tables created in the namespace after the branch exists: not members in
v1.
They can be added explicitly.
- A member table dropped or renamed while the branch is active: promotion
fails
with a clear error. Dropping the branch still works.
- Schema changes made on the branch: carried by the branch snapshots, so
promoting them is part of the same atomic switch.
- Concurrent promotions of two branches that touch the same table: the
second one
fails the fast-forward check for that table, so nothing is promoted twice.
- Federated and static-facade catalogs: not supported, same as
commitTransaction
today.

5. Out of scope for a first version

- Merging histories that have diverged (rebase or cherry-pick across
tables).
- Tags, commit logs and time travel at the catalog level, Nessie style.
- Adding this to the Iceberg REST spec. If it proves useful, I'd be glad to
propose a generalized version to the Iceberg community later.

6. Questions for the community

1. Atomicity across backends: promotion depends on a multi-entity
compare-and-set. Does every persistence backend (relational JDBC, NoSQL,
DynamoDB) give the same guarantee here, or does a backend need to advertise
support?
2. How large should a single promotion be allowed to get? With thousands of
tables, the transaction gets big. Should there be a limit, or a two-phase
approach?
3. Should Polaris protect a catalog branch's refs from direct deletion
through
the REST API?
4. Is a new entity type the right home, or would you prefer a property-based
approach on namespaces?
5. Any appetite for aligning the vocabulary with Nessie's, given the earlier
discussions about the two projects?

I'm happy to present this at the next community sync and to do the
implementation in small PRs: entity and persistence first, then create and
drop,
then promote, then the CLI and docs.

Thanks,
Ankur Kotwal

Re: [DISCUSS] Catalog-level branches for atomic multi-table promotion (#3469)

[Ankur Kotwal <ko...@gmail.com>]

Hi all,

As promised, here's a design doc for catalog-level branches (option C), open
for comments:

https://www.google.com/url?q=https://docs.google.com/document/d/1jLFkivxRZ0i4VHI-7G8olLQXk1sGAkUIFz_1xLGEJRA/edit&source=gmail&ust=1791530042497000&sa=E

The short version, so the main points are in the archive too:

- A branch is a second named pointer, cat/<id>/heads/<branch>, next to
heads/main, plus an entry in a new per-catalog registry (cat/<id>/branches),
since references can't be listed. Creating a branch, and publishing it while
main hasn't moved, are each one compare-and-swap, however many tables the
catalog has.
- Clients pick a branch through the prefix (warehouse "lake@backfill",
resolved
by CatalogPrefixParser) or a Polaris-Branch header, so engines don't change.
Catalog names can already contain "@", so a full-name match wins.
- Version 1 publishes only when main hasn't moved; merging comes later.
- Cleanup is the part that decides safety. Polaris objects: maintenance
walks
every registered pointer. Iceberg files: while a table is on more than one
branch, Polaris rejects remove-snapshots commits (REMOVE_TABLE_SNAPSHOTS),
sets gc.enabled=false once the table diverges (Iceberg's expiry and
orphan-file actions refuse to run then), and skips purge on drop. Later, a
catalog-side cleanup service enforcing the orphan-file-removal policy across
all branches replaces that, as Dmitri suggested.
- It needs the NoSQL model; with NoSQL on JDBC, RDBMS users get it too. On
the
classic JDBC persistence, the table-branch recipe from #5740 remains the
way.
- The plan is six small PRs behind a feature flag, starting with making the
persistence branch-aware with no behavior change.

The questions I'd most like input on are at the end of the doc, in
particular whether blocking destructive maintenance on branched tables is
acceptable for a first version.

If it's useful, I'm happy to walk through it at the community sync on
October 15.

Thanks,
Ankur

On Wed, Oct 07, 2026 11:42 PM, Ankur Kotwal <ko...@gmail.com> wrote:

> Hi Dmitri,
>
> Thanks, that's very helpful.
>
> Option B: understood, and thanks for the pointer to the multi-entity
> thread.
> The recipe only moves the refs of the tables it commits, and each of those
> is
> written with a check against the version that was read, so it doesn't
> depend on
> validations involving other entities. I've opened a test + doc PR as you
> suggested: https://www.google.com/url?q=https://github.com/apache/
> polaris/pull/5740&source=gmail&ust=1791528160935000&sa=E. Happy to add
> the same
> test against the NoSQL persistence if that helps.
>
> Option C: I'm glad to explore it from the start, and NoSQL on JDBC would
> make it
> much more broadly useful. Since GC is the part that decides whether
> branches are
> safe, I'd like to start there. My current thinking:
>
> - Polaris objects: branches need a registry the maintenance service can
> walk,
> since references can't be listed, and every head is retained with the same
> CEL rules as main.
> - Iceberg files: until there is a catalog-side maintenance service, a
> table that
> exists on more than one head gets gc.enabled=false. Iceberg's
> RemoveSnapshots
> and the Spark orphan-file action both refuse to run then ("GC is disabled
> (deleting files may corrupt other tables)"), and Polaris could reject
> snapshot-removing commits on those tables as a second guard. Later, a
> catalog-side service enforcing the existing orphan-file-removal policy
> across
> all heads would replace that, which matches your point about GC becoming a
> catalog concern.
>
> I'll write this up as a short design doc, covering branch selection (prefix
> and header) too, and share it here. If it's useful, I can walk through it
> at the
> next community sync.
>
> Thanks,
> Ankur
>
> On Wed, Oct 07, 2026 06:30 PM, Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
>> Hi Ankur,
>>
>> Thanks again for taking this feature discussion to the dev ML!
>>
>> Re: Option B: atomic multi-entity changes have some issues with the
>> current
>> Persistence framework. Basic operations work well, but there are concerns
>> with finer points. There is a rather long discussion about that in [1].
>>
>> I need some time to consider this idea. Let's see what other people think.
>>
>> If you could contribute a test + doc PR to Polaris for this idea, it would
>> be great!
>>
>> Re: Option C: Catalog-level branches are indeed natural in the NoSQL
>> Persistence. I'd personally prefer to explore this option from the start
>> (but I do not insist).
>>
>> If there's interest in supporting the NoSQL model over JDBC, I believe it
>> is quite possible to implement (despite the NoSQL name). That will open
>> native catalog branches to RDBMS-based environments too.
>>
>> The only operational drawback would be having to run NoSQL repository
>> maintenance [2] even on RDBMS, but I think it is quite an acceptable
>> inconvenience for the power of the feature as a whole.
>>
>> The traditional JDBC persistence and the proposed "Atomic DynamoDB" [5615]
>> persistence will remain as the "classical" options, of course. Users
>> interested in catalog-level branches will have the option to use NoSQL
>> with
>> MongoDB (current codebase), DynamoDB [5725] or JDBC (future PR).
>>
>> I certainly consider the NoSQL Persistence framework to be a long-term
>> feature. Specific backend support depends on user interest, I think, but
>> the range of possibilities virtually covers the majority of existing
>> databases. I'd also add that onboarding a new backend is rather simple.
>> All
>> the hard tasks are handled in common code.
>>
>> Re: Question 1: both the "prefix" and the "header" approaches can work, I
>> think. Even at the same time.
>>
>> Re: Question 2 (GC): IMHO, GC becomes a catalog-side concern, and that is
>> a
>> good thing. Polaris already has some relevant concepts,
>> e.g. OrphanFileRemovalPolicyContent [1261].
>>
>> IIRC, there were some prior discussions about an external "table
>> maintenance service" that could enforce those policies (I could not find a
>> specific dev thread, though).
>>
>> [1] https://lists.apache.org/thread/622d0sn4wzhw993dr5yhmz5zhy4228hp
>>
>> [2]
>> https://polaris.apache.org/releases/1.8.0/metastores/nosql-m
>> ongodb/#regular-maintenance
>>
>> [1261] https://github.com/apache/polaris/pull/1261
>>
>> [5615] https://github.com/apache/polaris/issues/5615
>>
>> [5725] https://github.com/apache/polaris/pull/5725
>>
>> Cheers,
>> Dmitri.
>>
>> On Wed, Oct 7, 2026 at 2:43 PM Ankur Kotwal <ko...@gmail.com>
>> wrote:
>>
>> > Hi all,
>> >
>> > Following up on my own proposal, since Dmitri questioned the new entity
>> > type on
>> > the issue and he had a point. I spent some time in the code, and there
>> are
>> > two
>> > things already in Polaris that I underestimated. Here's how I'd see the
>> > options
>> > now.
>> >
>> > What exists today
>> >
>> > 1. commitTransaction is atomic on both persistence families. The handler
>> > stages
>> > every table change and then writes all entities in one go: relational
>> JDBC
>> > in
>> > one database transaction with version checks, NoSQL as one commit
>> against
>> > the
>> > catalog's named pointer.
>> >
>> > 2. The NoSQL persistence is already shaped like Git: a catalog's
>> content is
>> > an
>> > immutable state object behind a named pointer, changes are commits with
>> a
>> > single CAS, and there's a commit log. The pointer is literally called
>> > cat/<id>/heads/main.
>> >
>> > Three ways to get to "promote many tables atomically"
>> >
>> > A. My original proposal: a CATALOG_BRANCH entity on top of per-table
>> > Iceberg
>> > branch refs. Looking at it again, the entity mostly duplicates what the
>> > refs
>> > already say, and promotion still touches every member table. I'd drop
>> it.
>> >
>> > B. No new server concept at all. A client creates a branch ref on each
>> > table in
>> > one transactions/commit, writes to it as usual, and promotes with one
>> more
>> > transactions/commit that, per table, asserts main is still at its base
>> > snapshot (assert-ref-snapshot-id) and moves main to the branch head
>> > (set-snapshot-ref). Either every table moves or none does, on every
>> > backend,
>> > today. What it can't do comes from Iceberg table branches: schemas are
>> per
>> > table, so a schema change on the branch is visible on main; tables,
>> views
>> > and
>> > namespaces can't be created or dropped "on a branch"; and the client
>> has to
>> > know the member list.
>> >
>> > I checked that this works with a small test against the REST handler
>> > (the in-memory metastore the service tests use): three tables, branch
>> > created in one commit, written to, promoted in one commit. When main of
>> one
>> > table was moved by another writer after branching, the promotion was
>> > rejected and none of the three tables changed. Happy to turn it into a
>> > proper test and a docs page.
>> >
>> > C. Real catalog branches as more named pointers,
>> cat/<id>/heads/<branch>,
>> > on the
>> > NoSQL persistence. Creating a branch is one new pointer; promoting is a
>> > single
>> > CAS when main hasn't moved, for any number of tables; schema changes and
>> > created or dropped tables live on the branch; and it's the same model as
>> > Nessie. It only works on NoSQL, and it needs real work: the reference
>> name
>> > threaded through the places that assume main (NoSqlMetaStore, References
>> > and
>> > the Table/Namespace/Policy mappings), a way for clients to pick a
>> branch,
>> > and
>> > above all garbage collection. Today CatalogRetainedIdentifier only
>> retains
>> > what heads/main reaches, and Iceberg snapshot expiry or orphan-file
>> removal
>> > run on main can delete files that only a branch's table metadata
>> > references.
>> > Nessie needed a dedicated GC for this, so I'd want that settled before
>> any
>> > code.
>> >
>> > What I'd suggest
>> >
>> > - Do B now: a documented recipe plus "polaris branch
>> create/promote/drop"
>> > commands in the CLI. No persistence or REST changes, works on every
>> > backend,
>> > and users get atomic multi-table promotion in the next release. It also
>> > tells
>> > us which of B's limits people actually hit.
>> > - Design C here on the list, starting with the two questions that decide
>> > the
>> > rest:
>> > 1. How does a client select a branch so engines don't change? A
>> warehouse
>> > suffix like "sales@backfill", a prefix segment from /v1/config, or a
>> > header?
>> > 2. How do we keep GC safe with several heads?
>> > And a broader one: is NoSQL the intended long-term default persistence?
>> A
>> > NoSQL-only feature is much easier to accept if it is.
>> >
>> > I wrote a longer comparison with the details and a table, and can share
>> it
>> > as a
>> > doc if that's useful. If B sounds reasonable, I can start on the CLI
>> part
>> > and
>> > the docs while we work out C.
>> >
>> > Thanks,
>> > Ankur
>> >
>> > On Tue, Oct 06, 2026 11:30 PM, Ankur Kotwal <ko...@gmail.com>
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > I'd like to start a discussion on catalog-level branching, tracked in
>> > > https://www.google.com/url?q=https://github.com/apache/
>> > > polaris/issues/3469&source=gmail&ust=1791440827911000&sa=E. Several
>> users
>> > > have asked for it
>> > > there, and Dmitri suggested taking it to the list. Below is a first
>> > > proposal. I'd
>> > > much rather shape it with you than arrive with a finished design, so
>> > > please poke
>> > > holes in it.
>> > >
>> > > 1. The problem
>> > >
>> > > Iceberg already lets you branch a single table: a branch is just
>> another
>> > > snapshot
>> > > ref in the table metadata. That works well for one table, but teams
>> that
>> > > run
>> > > backfills, migrations or "write-audit-publish" pipelines across many
>> > > tables hit
>> > > three walls:
>> > >
>> > > - Creating or merging a branch across N tables takes N separate calls.
>> > > - Promotion isn't atomic. While tables are merged one by one, readers
>> can
>> > > see
>> > > some tables with the new data and others without it. With hundreds of
>> > > tables,
>> > > that window is long, and a failure halfway through leaves the catalog
>> in
>> > a
>> > > mixed
>> > > state someone has to repair by hand.
>> > > - There's no cheap way to ask "which tables have branch X?" without
>> > > loading every
>> > > table's metadata.
>> > >
>> > > Today the answer is either to adopt Nessie, or to script it and accept
>> > the
>> > > non-atomic window.
>> > >
>> > > 2. What I'm proposing
>> > >
>> > > A catalog branch: a named, Polaris-managed group of per-table Iceberg
>> > > branches
>> > > that is created and promoted as a unit.
>> > >
>> > > polaris branch create backfill-2026 --from main --namespace sales
>> > > ... jobs write to the "backfill-2026" branch of each table as usual
>> ...
>> > > polaris branch promote backfill-2026 --into main # all tables, or none
>> > >
>> > > Design goals:
>> > >
>> > > - Engines don't change. Writers and readers use the standard Iceberg
>> > > branch API
>> > > (for example sales.orders.branch_backfill-2026 in Spark, or the "ref"
>> > > field
>> > > of REST commits). Nothing new is needed in Spark, Flink, Trino or
>> > > PyIceberg.
>> > > - Build on what Polaris already has. Per-table state stays in Iceberg
>> > > metadata,
>> > > and promotion reuses the transaction workspace that commitTransaction
>> > > already
>> > > uses to apply changes to several tables as one atomic metastore
>> update.
>> > > - Keep the first version small and safe. Fast-forward promotion only,
>> > > explicit
>> > > failure on conflict, no merge of diverged histories.
>> > >
>> > > 3. How it would work
>> > >
>> > > 3.1 A new entity
>> > >
>> > > A CATALOG_BRANCH entity, scoped to a catalog, holding:
>> > >
>> > > - the branch name,
>> > > - its scope (a namespace, optionally recursive, or an explicit list of
>> > > tables),
>> > > - for each member table: the table id and the snapshot id of main when
>> > the
>> > > branch was created (the base),
>> > > - created by / created at, and a state (ACTIVE, PROMOTED, DROPPED).
>> > >
>> > > Keeping membership in the entity answers "which tables have branch X?"
>> > > with one
>> > > read, instead of loading every table's metadata.
>> > >
>> > > 3.2 Create
>> > >
>> > > For every table in scope, add an Iceberg branch ref pointing at that
>> > > table's
>> > > current main snapshot, and write the CATALOG_BRANCH entity. All of it
>> > goes
>> > > through the transaction workspace, so either every table gets the ref
>> and
>> > > the
>> > > entity exists, or nothing changes.
>> > >
>> > > 3.3 Write
>> > >
>> > > Nothing new. Jobs commit to the table's branch ref through the normal
>> > REST
>> > > updateTable path. Polaris only needs to reject commits that would
>> move a
>> > > catalog branch's ref outside of Polaris' control in ways that break
>> > > promotion
>> > > (for example deleting the ref). Whether that check is worth having in
>> v1
>> > > is one
>> > > of the questions below.
>> > >
>> > > 3.4 Promote
>> > >
>> > > For every member table, in one transaction:
>> > >
>> > > 1. load the current metadata,
>> > > 2. check that main still points at the recorded base snapshot
>> > (fast-forward
>> > > check),
>> > > 3. set main to the branch head,
>> > > 4. optionally remove the branch ref.
>> > >
>> > > If any table fails the check, nothing is promoted, and the response
>> lists
>> > > the
>> > > tables that moved since the branch was created. Readers either see the
>> > old
>> > > state
>> > > of every table or the new state of every table.
>> > >
>> > > 3.5 Drop
>> > >
>> > > Remove the branch refs from member tables and mark the entity DROPPED.
>> > > Snapshot
>> > > expiry then cleans up the unreferenced data as it does today.
>> > >
>> > > 3.6 API and authorization
>> > >
>> > > A Polaris REST extension under the catalog API (create, list, get,
>> > > promote, drop),
>> > > plus CLI commands. Two options for authorization: new privileges
>> > > (CATALOG_BRANCH_CREATE, CATALOG_BRANCH_PROMOTE), or requiring
>> > > TABLE_WRITE_DATA
>> > > on every member table. The second is simpler, but slow to check for
>> very
>> > > large
>> > > branches.
>> > >
>> > > 4. Edge cases I've thought about so far
>> > >
>> > > - Tables created in the namespace after the branch exists: not
>> members in
>> > > v1.
>> > > They can be added explicitly.
>> > > - A member table dropped or renamed while the branch is active:
>> promotion
>> > > fails
>> > > with a clear error. Dropping the branch still works.
>> > > - Schema changes made on the branch: carried by the branch snapshots,
>> so
>> > > promoting them is part of the same atomic switch.
>> > > - Concurrent promotions of two branches that touch the same table: the
>> > > second one
>> > > fails the fast-forward check for that table, so nothing is promoted
>> > twice.
>> > > - Federated and static-facade catalogs: not supported, same as
>> > > commitTransaction
>> > > today.
>> > >
>> > > 5. Out of scope for a first version
>> > >
>> > > - Merging histories that have diverged (rebase or cherry-pick across
>> > > tables).
>> > > - Tags, commit logs and time travel at the catalog level, Nessie
>> style.
>> > > - Adding this to the Iceberg REST spec. If it proves useful, I'd be
>> glad
>> > to
>> > > propose a generalized version to the Iceberg community later.
>> > >
>> > > 6. Questions for the community
>> > >
>> > > 1. Atomicity across backends: promotion depends on a multi-entity
>> > > compare-and-set. Does every persistence backend (relational JDBC,
>> NoSQL,
>> > > DynamoDB) give the same guarantee here, or does a backend need to
>> > advertise
>> > > support?
>> > > 2. How large should a single promotion be allowed to get? With
>> thousands
>> > of
>> > > tables, the transaction gets big. Should there be a limit, or a
>> two-phase
>> > > approach?
>> > > 3. Should Polaris protect a catalog branch's refs from direct deletion
>> > > through
>> > > the REST API?
>> > > 4. Is a new entity type the right home, or would you prefer a
>> > > property-based
>> > > approach on namespaces?
>> > > 5. Any appetite for aligning the vocabulary with Nessie's, given the
>> > > earlier
>> > > discussions about the two projects?
>> > >
>> > > I'm happy to present this at the next community sync and to do the
>> > > implementation in small PRs: entity and persistence first, then create
>> > and
>> > > drop,
>> > > then promote, then the CLI and docs.
>> > >
>> > > Thanks,
>> > > Ankur Kotwal
>> > >
>> >
>>
>

Re: Re: [DISCUSS] Catalog-level branches for atomic multi-table promotion (#3469)

[Ankur Kotwal <ko...@gmail.com>]

Hi Dmitri,

Thanks, that's very helpful.

Option B: understood, and thanks for the pointer to the multi-entity thread.
The recipe only moves the refs of the tables it commits, and each of those
is
written with a check against the version that was read, so it doesn't
depend on
validations involving other entities. I've opened a test + doc PR as you
suggested:
https://www.google.com/url?q=https://github.com/apache/polaris/pull/5740&source=gmail&ust=1791528160935000&sa=E.
Happy to add the same
test against the NoSQL persistence if that helps.

Option C: I'm glad to explore it from the start, and NoSQL on JDBC would
make it
much more broadly useful. Since GC is the part that decides whether
branches are
safe, I'd like to start there. My current thinking:

- Polaris objects: branches need a registry the maintenance service can
walk,
since references can't be listed, and every head is retained with the same
CEL rules as main.
- Iceberg files: until there is a catalog-side maintenance service, a table
that
exists on more than one head gets gc.enabled=false. Iceberg's
RemoveSnapshots
and the Spark orphan-file action both refuse to run then ("GC is disabled
(deleting files may corrupt other tables)"), and Polaris could reject
snapshot-removing commits on those tables as a second guard. Later, a
catalog-side service enforcing the existing orphan-file-removal policy
across
all heads would replace that, which matches your point about GC becoming a
catalog concern.

I'll write this up as a short design doc, covering branch selection (prefix
and header) too, and share it here. If it's useful, I can walk through it
at the
next community sync.

Thanks,
Ankur

On Wed, Oct 07, 2026 06:30 PM, Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi Ankur,
>
> Thanks again for taking this feature discussion to the dev ML!
>
> Re: Option B: atomic multi-entity changes have some issues with the current
> Persistence framework. Basic operations work well, but there are concerns
> with finer points. There is a rather long discussion about that in [1].
>
> I need some time to consider this idea. Let's see what other people think.
>
> If you could contribute a test + doc PR to Polaris for this idea, it would
> be great!
>
> Re: Option C: Catalog-level branches are indeed natural in the NoSQL
> Persistence. I'd personally prefer to explore this option from the start
> (but I do not insist).
>
> If there's interest in supporting the NoSQL model over JDBC, I believe it
> is quite possible to implement (despite the NoSQL name). That will open
> native catalog branches to RDBMS-based environments too.
>
> The only operational drawback would be having to run NoSQL repository
> maintenance [2] even on RDBMS, but I think it is quite an acceptable
> inconvenience for the power of the feature as a whole.
>
> The traditional JDBC persistence and the proposed "Atomic DynamoDB" [5615]
> persistence will remain as the "classical" options, of course. Users
> interested in catalog-level branches will have the option to use NoSQL with
> MongoDB (current codebase), DynamoDB [5725] or JDBC (future PR).
>
> I certainly consider the NoSQL Persistence framework to be a long-term
> feature. Specific backend support depends on user interest, I think, but
> the range of possibilities virtually covers the majority of existing
> databases. I'd also add that onboarding a new backend is rather simple. All
> the hard tasks are handled in common code.
>
> Re: Question 1: both the "prefix" and the "header" approaches can work, I
> think. Even at the same time.
>
> Re: Question 2 (GC): IMHO, GC becomes a catalog-side concern, and that is a
> good thing. Polaris already has some relevant concepts,
> e.g. OrphanFileRemovalPolicyContent [1261].
>
> IIRC, there were some prior discussions about an external "table
> maintenance service" that could enforce those policies (I could not find a
> specific dev thread, though).
>
> [1] https://lists.apache.org/thread/622d0sn4wzhw993dr5yhmz5zhy4228hp
>
> [2]
> https://polaris.apache.org/releases/1.8.0/metastores/nosql-
> mongodb/#regular-maintenance
>
> [1261] https://github.com/apache/polaris/pull/1261
>
> [5615] https://github.com/apache/polaris/issues/5615
>
> [5725] https://github.com/apache/polaris/pull/5725
>
> Cheers,
> Dmitri.
>
> On Wed, Oct 7, 2026 at 2:43 PM Ankur Kotwal <ko...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > Following up on my own proposal, since Dmitri questioned the new entity
> > type on
> > the issue and he had a point. I spent some time in the code, and there
> are
> > two
> > things already in Polaris that I underestimated. Here's how I'd see the
> > options
> > now.
> >
> > What exists today
> >
> > 1. commitTransaction is atomic on both persistence families. The handler
> > stages
> > every table change and then writes all entities in one go: relational
> JDBC
> > in
> > one database transaction with version checks, NoSQL as one commit against
> > the
> > catalog's named pointer.
> >
> > 2. The NoSQL persistence is already shaped like Git: a catalog's content
> is
> > an
> > immutable state object behind a named pointer, changes are commits with a
> > single CAS, and there's a commit log. The pointer is literally called
> > cat/<id>/heads/main.
> >
> > Three ways to get to "promote many tables atomically"
> >
> > A. My original proposal: a CATALOG_BRANCH entity on top of per-table
> > Iceberg
> > branch refs. Looking at it again, the entity mostly duplicates what the
> > refs
> > already say, and promotion still touches every member table. I'd drop it.
> >
> > B. No new server concept at all. A client creates a branch ref on each
> > table in
> > one transactions/commit, writes to it as usual, and promotes with one
> more
> > transactions/commit that, per table, asserts main is still at its base
> > snapshot (assert-ref-snapshot-id) and moves main to the branch head
> > (set-snapshot-ref). Either every table moves or none does, on every
> > backend,
> > today. What it can't do comes from Iceberg table branches: schemas are
> per
> > table, so a schema change on the branch is visible on main; tables, views
> > and
> > namespaces can't be created or dropped "on a branch"; and the client has
> to
> > know the member list.
> >
> > I checked that this works with a small test against the REST handler
> > (the in-memory metastore the service tests use): three tables, branch
> > created in one commit, written to, promoted in one commit. When main of
> one
> > table was moved by another writer after branching, the promotion was
> > rejected and none of the three tables changed. Happy to turn it into a
> > proper test and a docs page.
> >
> > C. Real catalog branches as more named pointers, cat/<id>/heads/<branch>,
> > on the
> > NoSQL persistence. Creating a branch is one new pointer; promoting is a
> > single
> > CAS when main hasn't moved, for any number of tables; schema changes and
> > created or dropped tables live on the branch; and it's the same model as
> > Nessie. It only works on NoSQL, and it needs real work: the reference
> name
> > threaded through the places that assume main (NoSqlMetaStore, References
> > and
> > the Table/Namespace/Policy mappings), a way for clients to pick a branch,
> > and
> > above all garbage collection. Today CatalogRetainedIdentifier only
> retains
> > what heads/main reaches, and Iceberg snapshot expiry or orphan-file
> removal
> > run on main can delete files that only a branch's table metadata
> > references.
> > Nessie needed a dedicated GC for this, so I'd want that settled before
> any
> > code.
> >
> > What I'd suggest
> >
> > - Do B now: a documented recipe plus "polaris branch create/promote/drop"
> > commands in the CLI. No persistence or REST changes, works on every
> > backend,
> > and users get atomic multi-table promotion in the next release. It also
> > tells
> > us which of B's limits people actually hit.
> > - Design C here on the list, starting with the two questions that decide
> > the
> > rest:
> > 1. How does a client select a branch so engines don't change? A warehouse
> > suffix like "sales@backfill", a prefix segment from /v1/config, or a
> > header?
> > 2. How do we keep GC safe with several heads?
> > And a broader one: is NoSQL the intended long-term default persistence? A
> > NoSQL-only feature is much easier to accept if it is.
> >
> > I wrote a longer comparison with the details and a table, and can share
> it
> > as a
> > doc if that's useful. If B sounds reasonable, I can start on the CLI part
> > and
> > the docs while we work out C.
> >
> > Thanks,
> > Ankur
> >
> > On Tue, Oct 06, 2026 11:30 PM, Ankur Kotwal <ko...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start a discussion on catalog-level branching, tracked in
> > > https://www.google.com/url?q=https://github.com/apache/
> > > polaris/issues/3469&source=gmail&ust=1791440827911000&sa=E. Several
> users
> > > have asked for it
> > > there, and Dmitri suggested taking it to the list. Below is a first
> > > proposal. I'd
> > > much rather shape it with you than arrive with a finished design, so
> > > please poke
> > > holes in it.
> > >
> > > 1. The problem
> > >
> > > Iceberg already lets you branch a single table: a branch is just
> another
> > > snapshot
> > > ref in the table metadata. That works well for one table, but teams
> that
> > > run
> > > backfills, migrations or "write-audit-publish" pipelines across many
> > > tables hit
> > > three walls:
> > >
> > > - Creating or merging a branch across N tables takes N separate calls.
> > > - Promotion isn't atomic. While tables are merged one by one, readers
> can
> > > see
> > > some tables with the new data and others without it. With hundreds of
> > > tables,
> > > that window is long, and a failure halfway through leaves the catalog
> in
> > a
> > > mixed
> > > state someone has to repair by hand.
> > > - There's no cheap way to ask "which tables have branch X?" without
> > > loading every
> > > table's metadata.
> > >
> > > Today the answer is either to adopt Nessie, or to script it and accept
> > the
> > > non-atomic window.
> > >
> > > 2. What I'm proposing
> > >
> > > A catalog branch: a named, Polaris-managed group of per-table Iceberg
> > > branches
> > > that is created and promoted as a unit.
> > >
> > > polaris branch create backfill-2026 --from main --namespace sales
> > > ... jobs write to the "backfill-2026" branch of each table as usual ...
> > > polaris branch promote backfill-2026 --into main # all tables, or none
> > >
> > > Design goals:
> > >
> > > - Engines don't change. Writers and readers use the standard Iceberg
> > > branch API
> > > (for example sales.orders.branch_backfill-2026 in Spark, or the "ref"
> > > field
> > > of REST commits). Nothing new is needed in Spark, Flink, Trino or
> > > PyIceberg.
> > > - Build on what Polaris already has. Per-table state stays in Iceberg
> > > metadata,
> > > and promotion reuses the transaction workspace that commitTransaction
> > > already
> > > uses to apply changes to several tables as one atomic metastore update.
> > > - Keep the first version small and safe. Fast-forward promotion only,
> > > explicit
> > > failure on conflict, no merge of diverged histories.
> > >
> > > 3. How it would work
> > >
> > > 3.1 A new entity
> > >
> > > A CATALOG_BRANCH entity, scoped to a catalog, holding:
> > >
> > > - the branch name,
> > > - its scope (a namespace, optionally recursive, or an explicit list of
> > > tables),
> > > - for each member table: the table id and the snapshot id of main when
> > the
> > > branch was created (the base),
> > > - created by / created at, and a state (ACTIVE, PROMOTED, DROPPED).
> > >
> > > Keeping membership in the entity answers "which tables have branch X?"
> > > with one
> > > read, instead of loading every table's metadata.
> > >
> > > 3.2 Create
> > >
> > > For every table in scope, add an Iceberg branch ref pointing at that
> > > table's
> > > current main snapshot, and write the CATALOG_BRANCH entity. All of it
> > goes
> > > through the transaction workspace, so either every table gets the ref
> and
> > > the
> > > entity exists, or nothing changes.
> > >
> > > 3.3 Write
> > >
> > > Nothing new. Jobs commit to the table's branch ref through the normal
> > REST
> > > updateTable path. Polaris only needs to reject commits that would move
> a
> > > catalog branch's ref outside of Polaris' control in ways that break
> > > promotion
> > > (for example deleting the ref). Whether that check is worth having in
> v1
> > > is one
> > > of the questions below.
> > >
> > > 3.4 Promote
> > >
> > > For every member table, in one transaction:
> > >
> > > 1. load the current metadata,
> > > 2. check that main still points at the recorded base snapshot
> > (fast-forward
> > > check),
> > > 3. set main to the branch head,
> > > 4. optionally remove the branch ref.
> > >
> > > If any table fails the check, nothing is promoted, and the response
> lists
> > > the
> > > tables that moved since the branch was created. Readers either see the
> > old
> > > state
> > > of every table or the new state of every table.
> > >
> > > 3.5 Drop
> > >
> > > Remove the branch refs from member tables and mark the entity DROPPED.
> > > Snapshot
> > > expiry then cleans up the unreferenced data as it does today.
> > >
> > > 3.6 API and authorization
> > >
> > > A Polaris REST extension under the catalog API (create, list, get,
> > > promote, drop),
> > > plus CLI commands. Two options for authorization: new privileges
> > > (CATALOG_BRANCH_CREATE, CATALOG_BRANCH_PROMOTE), or requiring
> > > TABLE_WRITE_DATA
> > > on every member table. The second is simpler, but slow to check for
> very
> > > large
> > > branches.
> > >
> > > 4. Edge cases I've thought about so far
> > >
> > > - Tables created in the namespace after the branch exists: not members
> in
> > > v1.
> > > They can be added explicitly.
> > > - A member table dropped or renamed while the branch is active:
> promotion
> > > fails
> > > with a clear error. Dropping the branch still works.
> > > - Schema changes made on the branch: carried by the branch snapshots,
> so
> > > promoting them is part of the same atomic switch.
> > > - Concurrent promotions of two branches that touch the same table: the
> > > second one
> > > fails the fast-forward check for that table, so nothing is promoted
> > twice.
> > > - Federated and static-facade catalogs: not supported, same as
> > > commitTransaction
> > > today.
> > >
> > > 5. Out of scope for a first version
> > >
> > > - Merging histories that have diverged (rebase or cherry-pick across
> > > tables).
> > > - Tags, commit logs and time travel at the catalog level, Nessie style.
> > > - Adding this to the Iceberg REST spec. If it proves useful, I'd be
> glad
> > to
> > > propose a generalized version to the Iceberg community later.
> > >
> > > 6. Questions for the community
> > >
> > > 1. Atomicity across backends: promotion depends on a multi-entity
> > > compare-and-set. Does every persistence backend (relational JDBC,
> NoSQL,
> > > DynamoDB) give the same guarantee here, or does a backend need to
> > advertise
> > > support?
> > > 2. How large should a single promotion be allowed to get? With
> thousands
> > of
> > > tables, the transaction gets big. Should there be a limit, or a
> two-phase
> > > approach?
> > > 3. Should Polaris protect a catalog branch's refs from direct deletion
> > > through
> > > the REST API?
> > > 4. Is a new entity type the right home, or would you prefer a
> > > property-based
> > > approach on namespaces?
> > > 5. Any appetite for aligning the vocabulary with Nessie's, given the
> > > earlier
> > > discussions about the two projects?
> > >
> > > I'm happy to present this at the next community sync and to do the
> > > implementation in small PRs: entity and persistence first, then create
> > and
> > > drop,
> > > then promote, then the CLI and docs.
> > >
> > > Thanks,
> > > Ankur Kotwal
> > >
> >
>

Re: [DISCUSS] Catalog-level branches for atomic multi-table promotion (#3469)

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Ankur,

Thanks again for taking this feature discussion to the dev ML!

Re: Option B: atomic multi-entity changes have some issues with the current
Persistence framework. Basic operations work well, but there are concerns
with finer points. There is a rather long discussion about that in [1].

I need some time to consider this idea. Let's see what other people think.

If you could contribute a test + doc PR to Polaris for this idea, it would
be great!

Re: Option C: Catalog-level branches are indeed natural in the NoSQL
Persistence. I'd personally prefer to explore this option from the start
(but I do not insist).

If there's interest in supporting the NoSQL model over JDBC, I believe it
is quite possible to implement (despite the NoSQL name). That will open
native catalog branches to RDBMS-based environments too.

The only operational drawback would be having to run NoSQL repository
maintenance [2] even on RDBMS, but I think it is quite an acceptable
inconvenience for the power of the feature as a whole.

The traditional JDBC persistence and the proposed "Atomic DynamoDB" [5615]
persistence will remain as the "classical" options, of course. Users
interested in catalog-level branches will have the option to use NoSQL with
MongoDB (current codebase), DynamoDB [5725] or JDBC (future PR).

I certainly consider the NoSQL Persistence framework to be a long-term
feature. Specific backend support depends on user interest, I think, but
the range of possibilities virtually covers the majority of existing
databases. I'd also add that onboarding a new backend is rather simple. All
the hard tasks are handled in common code.

Re: Question 1: both the "prefix" and the "header" approaches can work, I
think. Even at the same time.

Re: Question 2 (GC): IMHO, GC becomes a catalog-side concern, and that is a
good thing. Polaris already has some relevant concepts,
e.g. OrphanFileRemovalPolicyContent [1261].

IIRC, there were some prior discussions about an external "table
maintenance service" that could enforce those policies (I could not find a
specific dev thread, though).

[1] https://lists.apache.org/thread/622d0sn4wzhw993dr5yhmz5zhy4228hp

[2]
https://polaris.apache.org/releases/1.8.0/metastores/nosql-mongodb/#regular-maintenance

[1261] https://github.com/apache/polaris/pull/1261

[5615] https://github.com/apache/polaris/issues/5615

[5725] https://github.com/apache/polaris/pull/5725

Cheers,
Dmitri.

On Wed, Oct 7, 2026 at 2:43 PM Ankur Kotwal <ko...@gmail.com> wrote:

> Hi all,
>
> Following up on my own proposal, since Dmitri questioned the new entity
> type on
> the issue and he had a point. I spent some time in the code, and there are
> two
> things already in Polaris that I underestimated. Here's how I'd see the
> options
> now.
>
> What exists today
>
> 1. commitTransaction is atomic on both persistence families. The handler
> stages
> every table change and then writes all entities in one go: relational JDBC
> in
> one database transaction with version checks, NoSQL as one commit against
> the
> catalog's named pointer.
>
> 2. The NoSQL persistence is already shaped like Git: a catalog's content is
> an
> immutable state object behind a named pointer, changes are commits with a
> single CAS, and there's a commit log. The pointer is literally called
> cat/<id>/heads/main.
>
> Three ways to get to "promote many tables atomically"
>
> A. My original proposal: a CATALOG_BRANCH entity on top of per-table
> Iceberg
> branch refs. Looking at it again, the entity mostly duplicates what the
> refs
> already say, and promotion still touches every member table. I'd drop it.
>
> B. No new server concept at all. A client creates a branch ref on each
> table in
> one transactions/commit, writes to it as usual, and promotes with one more
> transactions/commit that, per table, asserts main is still at its base
> snapshot (assert-ref-snapshot-id) and moves main to the branch head
> (set-snapshot-ref). Either every table moves or none does, on every
> backend,
> today. What it can't do comes from Iceberg table branches: schemas are per
> table, so a schema change on the branch is visible on main; tables, views
> and
> namespaces can't be created or dropped "on a branch"; and the client has to
> know the member list.
>
> I checked that this works with a small test against the REST handler
> (the in-memory metastore the service tests use): three tables, branch
> created in one commit, written to, promoted in one commit. When main of one
> table was moved by another writer after branching, the promotion was
> rejected and none of the three tables changed. Happy to turn it into a
> proper test and a docs page.
>
> C. Real catalog branches as more named pointers, cat/<id>/heads/<branch>,
> on the
> NoSQL persistence. Creating a branch is one new pointer; promoting is a
> single
> CAS when main hasn't moved, for any number of tables; schema changes and
> created or dropped tables live on the branch; and it's the same model as
> Nessie. It only works on NoSQL, and it needs real work: the reference name
> threaded through the places that assume main (NoSqlMetaStore, References
> and
> the Table/Namespace/Policy mappings), a way for clients to pick a branch,
> and
> above all garbage collection. Today CatalogRetainedIdentifier only retains
> what heads/main reaches, and Iceberg snapshot expiry or orphan-file removal
> run on main can delete files that only a branch's table metadata
> references.
> Nessie needed a dedicated GC for this, so I'd want that settled before any
> code.
>
> What I'd suggest
>
> - Do B now: a documented recipe plus "polaris branch create/promote/drop"
> commands in the CLI. No persistence or REST changes, works on every
> backend,
> and users get atomic multi-table promotion in the next release. It also
> tells
> us which of B's limits people actually hit.
> - Design C here on the list, starting with the two questions that decide
> the
> rest:
> 1. How does a client select a branch so engines don't change? A warehouse
> suffix like "sales@backfill", a prefix segment from /v1/config, or a
> header?
> 2. How do we keep GC safe with several heads?
> And a broader one: is NoSQL the intended long-term default persistence? A
> NoSQL-only feature is much easier to accept if it is.
>
> I wrote a longer comparison with the details and a table, and can share it
> as a
> doc if that's useful. If B sounds reasonable, I can start on the CLI part
> and
> the docs while we work out C.
>
> Thanks,
> Ankur
>
> On Tue, Oct 06, 2026 11:30 PM, Ankur Kotwal <ko...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion on catalog-level branching, tracked in
> > https://www.google.com/url?q=https://github.com/apache/
> > polaris/issues/3469&source=gmail&ust=1791440827911000&sa=E. Several users
> > have asked for it
> > there, and Dmitri suggested taking it to the list. Below is a first
> > proposal. I'd
> > much rather shape it with you than arrive with a finished design, so
> > please poke
> > holes in it.
> >
> > 1. The problem
> >
> > Iceberg already lets you branch a single table: a branch is just another
> > snapshot
> > ref in the table metadata. That works well for one table, but teams that
> > run
> > backfills, migrations or "write-audit-publish" pipelines across many
> > tables hit
> > three walls:
> >
> > - Creating or merging a branch across N tables takes N separate calls.
> > - Promotion isn't atomic. While tables are merged one by one, readers can
> > see
> > some tables with the new data and others without it. With hundreds of
> > tables,
> > that window is long, and a failure halfway through leaves the catalog in
> a
> > mixed
> > state someone has to repair by hand.
> > - There's no cheap way to ask "which tables have branch X?" without
> > loading every
> > table's metadata.
> >
> > Today the answer is either to adopt Nessie, or to script it and accept
> the
> > non-atomic window.
> >
> > 2. What I'm proposing
> >
> > A catalog branch: a named, Polaris-managed group of per-table Iceberg
> > branches
> > that is created and promoted as a unit.
> >
> > polaris branch create backfill-2026 --from main --namespace sales
> > ... jobs write to the "backfill-2026" branch of each table as usual ...
> > polaris branch promote backfill-2026 --into main # all tables, or none
> >
> > Design goals:
> >
> > - Engines don't change. Writers and readers use the standard Iceberg
> > branch API
> > (for example sales.orders.branch_backfill-2026 in Spark, or the "ref"
> > field
> > of REST commits). Nothing new is needed in Spark, Flink, Trino or
> > PyIceberg.
> > - Build on what Polaris already has. Per-table state stays in Iceberg
> > metadata,
> > and promotion reuses the transaction workspace that commitTransaction
> > already
> > uses to apply changes to several tables as one atomic metastore update.
> > - Keep the first version small and safe. Fast-forward promotion only,
> > explicit
> > failure on conflict, no merge of diverged histories.
> >
> > 3. How it would work
> >
> > 3.1 A new entity
> >
> > A CATALOG_BRANCH entity, scoped to a catalog, holding:
> >
> > - the branch name,
> > - its scope (a namespace, optionally recursive, or an explicit list of
> > tables),
> > - for each member table: the table id and the snapshot id of main when
> the
> > branch was created (the base),
> > - created by / created at, and a state (ACTIVE, PROMOTED, DROPPED).
> >
> > Keeping membership in the entity answers "which tables have branch X?"
> > with one
> > read, instead of loading every table's metadata.
> >
> > 3.2 Create
> >
> > For every table in scope, add an Iceberg branch ref pointing at that
> > table's
> > current main snapshot, and write the CATALOG_BRANCH entity. All of it
> goes
> > through the transaction workspace, so either every table gets the ref and
> > the
> > entity exists, or nothing changes.
> >
> > 3.3 Write
> >
> > Nothing new. Jobs commit to the table's branch ref through the normal
> REST
> > updateTable path. Polaris only needs to reject commits that would move a
> > catalog branch's ref outside of Polaris' control in ways that break
> > promotion
> > (for example deleting the ref). Whether that check is worth having in v1
> > is one
> > of the questions below.
> >
> > 3.4 Promote
> >
> > For every member table, in one transaction:
> >
> > 1. load the current metadata,
> > 2. check that main still points at the recorded base snapshot
> (fast-forward
> > check),
> > 3. set main to the branch head,
> > 4. optionally remove the branch ref.
> >
> > If any table fails the check, nothing is promoted, and the response lists
> > the
> > tables that moved since the branch was created. Readers either see the
> old
> > state
> > of every table or the new state of every table.
> >
> > 3.5 Drop
> >
> > Remove the branch refs from member tables and mark the entity DROPPED.
> > Snapshot
> > expiry then cleans up the unreferenced data as it does today.
> >
> > 3.6 API and authorization
> >
> > A Polaris REST extension under the catalog API (create, list, get,
> > promote, drop),
> > plus CLI commands. Two options for authorization: new privileges
> > (CATALOG_BRANCH_CREATE, CATALOG_BRANCH_PROMOTE), or requiring
> > TABLE_WRITE_DATA
> > on every member table. The second is simpler, but slow to check for very
> > large
> > branches.
> >
> > 4. Edge cases I've thought about so far
> >
> > - Tables created in the namespace after the branch exists: not members in
> > v1.
> > They can be added explicitly.
> > - A member table dropped or renamed while the branch is active: promotion
> > fails
> > with a clear error. Dropping the branch still works.
> > - Schema changes made on the branch: carried by the branch snapshots, so
> > promoting them is part of the same atomic switch.
> > - Concurrent promotions of two branches that touch the same table: the
> > second one
> > fails the fast-forward check for that table, so nothing is promoted
> twice.
> > - Federated and static-facade catalogs: not supported, same as
> > commitTransaction
> > today.
> >
> > 5. Out of scope for a first version
> >
> > - Merging histories that have diverged (rebase or cherry-pick across
> > tables).
> > - Tags, commit logs and time travel at the catalog level, Nessie style.
> > - Adding this to the Iceberg REST spec. If it proves useful, I'd be glad
> to
> > propose a generalized version to the Iceberg community later.
> >
> > 6. Questions for the community
> >
> > 1. Atomicity across backends: promotion depends on a multi-entity
> > compare-and-set. Does every persistence backend (relational JDBC, NoSQL,
> > DynamoDB) give the same guarantee here, or does a backend need to
> advertise
> > support?
> > 2. How large should a single promotion be allowed to get? With thousands
> of
> > tables, the transaction gets big. Should there be a limit, or a two-phase
> > approach?
> > 3. Should Polaris protect a catalog branch's refs from direct deletion
> > through
> > the REST API?
> > 4. Is a new entity type the right home, or would you prefer a
> > property-based
> > approach on namespaces?
> > 5. Any appetite for aligning the vocabulary with Nessie's, given the
> > earlier
> > discussions about the two projects?
> >
> > I'm happy to present this at the next community sync and to do the
> > implementation in small PRs: entity and persistence first, then create
> and
> > drop,
> > then promote, then the CLI and docs.
> >
> > Thanks,
> > Ankur Kotwal
> >
>

Re: [DISCUSS] Catalog-level branches for atomic multi-table promotion (#3469)

[Ankur Kotwal <ko...@gmail.com>]

Hi all,

Following up on my own proposal, since Dmitri questioned the new entity
type on
the issue and he had a point. I spent some time in the code, and there are
two
things already in Polaris that I underestimated. Here's how I'd see the
options
now.

What exists today

1. commitTransaction is atomic on both persistence families. The handler
stages
every table change and then writes all entities in one go: relational JDBC
in
one database transaction with version checks, NoSQL as one commit against
the
catalog's named pointer.

2. The NoSQL persistence is already shaped like Git: a catalog's content is
an
immutable state object behind a named pointer, changes are commits with a
single CAS, and there's a commit log. The pointer is literally called
cat/<id>/heads/main.

Three ways to get to "promote many tables atomically"

A. My original proposal: a CATALOG_BRANCH entity on top of per-table Iceberg
branch refs. Looking at it again, the entity mostly duplicates what the refs
already say, and promotion still touches every member table. I'd drop it.

B. No new server concept at all. A client creates a branch ref on each
table in
one transactions/commit, writes to it as usual, and promotes with one more
transactions/commit that, per table, asserts main is still at its base
snapshot (assert-ref-snapshot-id) and moves main to the branch head
(set-snapshot-ref). Either every table moves or none does, on every backend,
today. What it can't do comes from Iceberg table branches: schemas are per
table, so a schema change on the branch is visible on main; tables, views
and
namespaces can't be created or dropped "on a branch"; and the client has to
know the member list.

I checked that this works with a small test against the REST handler
(the in-memory metastore the service tests use): three tables, branch
created in one commit, written to, promoted in one commit. When main of one
table was moved by another writer after branching, the promotion was
rejected and none of the three tables changed. Happy to turn it into a
proper test and a docs page.

C. Real catalog branches as more named pointers, cat/<id>/heads/<branch>,
on the
NoSQL persistence. Creating a branch is one new pointer; promoting is a
single
CAS when main hasn't moved, for any number of tables; schema changes and
created or dropped tables live on the branch; and it's the same model as
Nessie. It only works on NoSQL, and it needs real work: the reference name
threaded through the places that assume main (NoSqlMetaStore, References and
the Table/Namespace/Policy mappings), a way for clients to pick a branch,
and
above all garbage collection. Today CatalogRetainedIdentifier only retains
what heads/main reaches, and Iceberg snapshot expiry or orphan-file removal
run on main can delete files that only a branch's table metadata references.
Nessie needed a dedicated GC for this, so I'd want that settled before any
code.

What I'd suggest

- Do B now: a documented recipe plus "polaris branch create/promote/drop"
commands in the CLI. No persistence or REST changes, works on every backend,
and users get atomic multi-table promotion in the next release. It also
tells
us which of B's limits people actually hit.
- Design C here on the list, starting with the two questions that decide the
rest:
1. How does a client select a branch so engines don't change? A warehouse
suffix like "sales@backfill", a prefix segment from /v1/config, or a
header?
2. How do we keep GC safe with several heads?
And a broader one: is NoSQL the intended long-term default persistence? A
NoSQL-only feature is much easier to accept if it is.

I wrote a longer comparison with the details and a table, and can share it
as a
doc if that's useful. If B sounds reasonable, I can start on the CLI part
and
the docs while we work out C.

Thanks,
Ankur

On Tue, Oct 06, 2026 11:30 PM, Ankur Kotwal <ko...@gmail.com> wrote:

> Hi all,
>
> I'd like to start a discussion on catalog-level branching, tracked in
> https://www.google.com/url?q=https://github.com/apache/
> polaris/issues/3469&source=gmail&ust=1791440827911000&sa=E. Several users
> have asked for it
> there, and Dmitri suggested taking it to the list. Below is a first
> proposal. I'd
> much rather shape it with you than arrive with a finished design, so
> please poke
> holes in it.
>
> 1. The problem
>
> Iceberg already lets you branch a single table: a branch is just another
> snapshot
> ref in the table metadata. That works well for one table, but teams that
> run
> backfills, migrations or "write-audit-publish" pipelines across many
> tables hit
> three walls:
>
> - Creating or merging a branch across N tables takes N separate calls.
> - Promotion isn't atomic. While tables are merged one by one, readers can
> see
> some tables with the new data and others without it. With hundreds of
> tables,
> that window is long, and a failure halfway through leaves the catalog in a
> mixed
> state someone has to repair by hand.
> - There's no cheap way to ask "which tables have branch X?" without
> loading every
> table's metadata.
>
> Today the answer is either to adopt Nessie, or to script it and accept the
> non-atomic window.
>
> 2. What I'm proposing
>
> A catalog branch: a named, Polaris-managed group of per-table Iceberg
> branches
> that is created and promoted as a unit.
>
> polaris branch create backfill-2026 --from main --namespace sales
> ... jobs write to the "backfill-2026" branch of each table as usual ...
> polaris branch promote backfill-2026 --into main # all tables, or none
>
> Design goals:
>
> - Engines don't change. Writers and readers use the standard Iceberg
> branch API
> (for example sales.orders.branch_backfill-2026 in Spark, or the "ref"
> field
> of REST commits). Nothing new is needed in Spark, Flink, Trino or
> PyIceberg.
> - Build on what Polaris already has. Per-table state stays in Iceberg
> metadata,
> and promotion reuses the transaction workspace that commitTransaction
> already
> uses to apply changes to several tables as one atomic metastore update.
> - Keep the first version small and safe. Fast-forward promotion only,
> explicit
> failure on conflict, no merge of diverged histories.
>
> 3. How it would work
>
> 3.1 A new entity
>
> A CATALOG_BRANCH entity, scoped to a catalog, holding:
>
> - the branch name,
> - its scope (a namespace, optionally recursive, or an explicit list of
> tables),
> - for each member table: the table id and the snapshot id of main when the
> branch was created (the base),
> - created by / created at, and a state (ACTIVE, PROMOTED, DROPPED).
>
> Keeping membership in the entity answers "which tables have branch X?"
> with one
> read, instead of loading every table's metadata.
>
> 3.2 Create
>
> For every table in scope, add an Iceberg branch ref pointing at that
> table's
> current main snapshot, and write the CATALOG_BRANCH entity. All of it goes
> through the transaction workspace, so either every table gets the ref and
> the
> entity exists, or nothing changes.
>
> 3.3 Write
>
> Nothing new. Jobs commit to the table's branch ref through the normal REST
> updateTable path. Polaris only needs to reject commits that would move a
> catalog branch's ref outside of Polaris' control in ways that break
> promotion
> (for example deleting the ref). Whether that check is worth having in v1
> is one
> of the questions below.
>
> 3.4 Promote
>
> For every member table, in one transaction:
>
> 1. load the current metadata,
> 2. check that main still points at the recorded base snapshot (fast-forward
> check),
> 3. set main to the branch head,
> 4. optionally remove the branch ref.
>
> If any table fails the check, nothing is promoted, and the response lists
> the
> tables that moved since the branch was created. Readers either see the old
> state
> of every table or the new state of every table.
>
> 3.5 Drop
>
> Remove the branch refs from member tables and mark the entity DROPPED.
> Snapshot
> expiry then cleans up the unreferenced data as it does today.
>
> 3.6 API and authorization
>
> A Polaris REST extension under the catalog API (create, list, get,
> promote, drop),
> plus CLI commands. Two options for authorization: new privileges
> (CATALOG_BRANCH_CREATE, CATALOG_BRANCH_PROMOTE), or requiring
> TABLE_WRITE_DATA
> on every member table. The second is simpler, but slow to check for very
> large
> branches.
>
> 4. Edge cases I've thought about so far
>
> - Tables created in the namespace after the branch exists: not members in
> v1.
> They can be added explicitly.
> - A member table dropped or renamed while the branch is active: promotion
> fails
> with a clear error. Dropping the branch still works.
> - Schema changes made on the branch: carried by the branch snapshots, so
> promoting them is part of the same atomic switch.
> - Concurrent promotions of two branches that touch the same table: the
> second one
> fails the fast-forward check for that table, so nothing is promoted twice.
> - Federated and static-facade catalogs: not supported, same as
> commitTransaction
> today.
>
> 5. Out of scope for a first version
>
> - Merging histories that have diverged (rebase or cherry-pick across
> tables).
> - Tags, commit logs and time travel at the catalog level, Nessie style.
> - Adding this to the Iceberg REST spec. If it proves useful, I'd be glad to
> propose a generalized version to the Iceberg community later.
>
> 6. Questions for the community
>
> 1. Atomicity across backends: promotion depends on a multi-entity
> compare-and-set. Does every persistence backend (relational JDBC, NoSQL,
> DynamoDB) give the same guarantee here, or does a backend need to advertise
> support?
> 2. How large should a single promotion be allowed to get? With thousands of
> tables, the transaction gets big. Should there be a limit, or a two-phase
> approach?
> 3. Should Polaris protect a catalog branch's refs from direct deletion
> through
> the REST API?
> 4. Is a new entity type the right home, or would you prefer a
> property-based
> approach on namespaces?
> 5. Any appetite for aligning the vocabulary with Nessie's, given the
> earlier
> discussions about the two projects?
>
> I'm happy to present this at the next community sync and to do the
> implementation in small PRs: entity and persistence first, then create and
> drop,
> then promote, then the CLI and docs.
>
> Thanks,
> Ankur Kotwal
>