[DISCUSS] Multi-table, multi-statement transactions in Polaris

[Qiegang Long <ql...@gmail.com>]

Hi all,

Thanks to everyone who came to the multi-table transactions talk at
Lakehouse Day in Glasgow, and for the conversations afterwards.
Several of you suggested continuing here. I'm new to the Polaris
community: I built this as a proof of concept on Polaris, and this is
the first time I'm bringing it to the list.

Iceberg already has most of the foundation for transactions engines
can use: the REST commitTransaction bundle commits several tables
atomically, and table branches can stage writes. What's missing is the
engine usable transaction around them: BEGIN ... COMMIT across
statements, with one consistent read point across tables. The
discussion of this on the Iceberg list has been quiet for a while, and
I built the PoC to restart it with something concrete.

The PoC runs Spark and Trino against a Polaris fork: multi-statement
transactions at READ COMMITTED and REPEATABLE READ, a read point
shared across engines, and fan-out writes from independent workers
that commit atomically. After conversations with Iceberg folks at the
conference, I'm moving the write path onto Iceberg table branches, so
writes become standard REST updates; I'll follow up in this thread
with the updated code.

For example, in Spark with the Polaris plugin:

    BEGIN TRANSACTION;
    INSERT INTO customers VALUES (22, 'joe', 'joe@example.com');
    INSERT INTO orders    VALUES (2002, 22, 9.99, 'pending');
    COMMIT;   -- both tables become visible together, or neither does

Readers in Spark or Trino, at either isolation level, never see the
order without its customer.

The change to the Iceberg REST surface is small: one optional
loadTable parameter (the read point). Everything else is Polaris-only
endpoints (minting a read point, fan-out coordination). So it could
live in Polaris first as an opt-in extension, with an Iceberg proposal
later if the community wants one.

Would Polaris be interested in carrying it as an extension first? And
since I'm new here, I'd appreciate guidance on the usual path for a
change of this size: for example a design review here first, an issue
to track it, time at a community sync, or going straight to small PRs.
I'm happy to work whichever way suits the project. It's an early PoC
with known gaps (listed in the design doc), and JDBC/PostgreSQL only
for now.

Talk, slides, demo video, design doc and code:
https://github.com/qlong/lakehouse-transactions

Thanks,
Qiegang

Re: [DISCUSS] Multi-table, multi-statement transactions in Polaris

[Qiegang Long <ql...@gmail.com>]

Hi Ankur, Yufei,

Thanks to both of you, and Ankur, sorry for the slow reply.

Yufei:  yes, that's the design doc:
https://docs.google.com/document/d/1ew2dOI_BHQFP1C8Q46Y_m_AvNNH4rzzLKaSsB089GTU/edit?tab=t.0
Reviewing it here first and splitting the implementation into small
PRs afterwards is the plan. Comments directly in the doc work too. The
earlier messages are in the archive, in case they went to spam:
https://lists.apache.org/thread/d5ft79ogcxx15ccr1n1bc0rvsl14tf6v

Ankur,  on your points:

1. Server-side publish (question 10): thanks, the missing cherry-pick
in PyIceberg is a good example of how hard the publish rules are to
keep the same across clients. The server already publishes fan-out
transactions this way, so offering it for single-session is mainly a
question of moving that work, and its rebase cost, onto the catalog
for the common case. Scalability, and leaving the client free to
resolve a conflict itself (a server can only re-validate a change, not
replay it), were the reasons the design publishes single-session
transactions on the client. Definitely open on this, IMHO it is a
critical decision point.  On the delete that matched no rows: in the
design, a branch that was never written is dropped without an
assertion on main. Under snapshot isolation that's correct, since such
a delete is a read. Under SERIALIZABLE it has to be validated like any
other read, which is part of the read-set validation listed as future
work.

2. #5740: thanks, I'll use BranchPromotionRecipeTest as the baseline
for the fast-forward case, so the recipe and the publish describe the
same commit.

3. Snapshot expiry (question 16): rejecting remove-snapshots only for
snapshots a registered read point still needs, and keeping 410 for
unregistered handles, sounds right to me.

4. Branch visibility (question 1): a reserved prefix is a good option
(the PoC already names staging branches with a txn_ prefix), will add
this along with alternatives.

I'll add these to the open questions in the doc.

Thanks,
Qiegang


On Fri, Nov 6, 2026 at 7:50 PM Yufei Gu <fl...@gmail.com> wrote:
>
> Hi Qiegang,
>
> Thanks for sharing this and putting together the PoC! Sorry for the
> late reply. This would be a great feature to have in Polaris as we
> discussed at Coc Glasgow.
>
> I'd suggest starting with a review of the design doc on the dev list
> so we can align on the scope, transaction semantics, and API changes.
> Once we agree on the design, we can break the implementation into
> smaller PRs. I believe this is the design doc, right?
> https://docs.google.com/document/d/1ew2dOI_BHQFP1C8Q46Y_m_AvNNH4rzzLKaSsB089GTU/edit?tab=t.0
>
> Sorry if I missed other discussions, some ASF mails have recently
> landed in my spam folder.
>
> Yufei
>
> On Mon, Oct 19, 2026 at 12:32 PM Qiegang Long <ql...@gmail.com> wrote:
> >
> > Hi all,
> >
> > Thanks to everyone who came to the multi-table transactions talk at
> > Lakehouse Day in Glasgow, and for the conversations afterwards.
> > Several of you suggested continuing here. I'm new to the Polaris
> > community: I built this as a proof of concept on Polaris, and this is
> > the first time I'm bringing it to the list.
> >
> > Iceberg already has most of the foundation for transactions engines
> > can use: the REST commitTransaction bundle commits several tables
> > atomically, and table branches can stage writes. What's missing is the
> > engine usable transaction around them: BEGIN ... COMMIT across
> > statements, with one consistent read point across tables. The
> > discussion of this on the Iceberg list has been quiet for a while, and
> > I built the PoC to restart it with something concrete.
> >
> > The PoC runs Spark and Trino against a Polaris fork: multi-statement
> > transactions at READ COMMITTED and REPEATABLE READ, a read point
> > shared across engines, and fan-out writes from independent workers
> > that commit atomically. After conversations with Iceberg folks at the
> > conference, I'm moving the write path onto Iceberg table branches, so
> > writes become standard REST updates; I'll follow up in this thread
> > with the updated code.
> >
> > For example, in Spark with the Polaris plugin:
> >
> >     BEGIN TRANSACTION;
> >     INSERT INTO customers VALUES (22, 'joe', 'joe@example.com');
> >     INSERT INTO orders    VALUES (2002, 22, 9.99, 'pending');
> >     COMMIT;   -- both tables become visible together, or neither does
> >
> > Readers in Spark or Trino, at either isolation level, never see the
> > order without its customer.
> >
> > The change to the Iceberg REST surface is small: one optional
> > loadTable parameter (the read point). Everything else is Polaris-only
> > endpoints (minting a read point, fan-out coordination). So it could
> > live in Polaris first as an opt-in extension, with an Iceberg proposal
> > later if the community wants one.
> >
> > Would Polaris be interested in carrying it as an extension first? And
> > since I'm new here, I'd appreciate guidance on the usual path for a
> > change of this size: for example a design review here first, an issue
> > to track it, time at a community sync, or going straight to small PRs.
> > I'm happy to work whichever way suits the project. It's an early PoC
> > with known gaps (listed in the design doc), and JDBC/PostgreSQL only
> > for now.
> >
> > Talk, slides, demo video, design doc and code:
> > https://github.com/qlong/lakehouse-transactions
> >
> > Thanks,
> > Qiegang

Re: [DISCUSS] Multi-table, multi-statement transactions in Polaris

[Yufei Gu <fl...@gmail.com>]

Hi Qiegang,

Thanks for sharing this and putting together the PoC! Sorry for the
late reply. This would be a great feature to have in Polaris as we
discussed at Coc Glasgow.

I'd suggest starting with a review of the design doc on the dev list
so we can align on the scope, transaction semantics, and API changes.
Once we agree on the design, we can break the implementation into
smaller PRs. I believe this is the design doc, right?
https://docs.google.com/document/d/1ew2dOI_BHQFP1C8Q46Y_m_AvNNH4rzzLKaSsB089GTU/edit?tab=t.0

Sorry if I missed other discussions, some ASF mails have recently
landed in my spam folder.

Yufei

On Mon, Oct 19, 2026 at 12:32 PM Qiegang Long <ql...@gmail.com> wrote:
>
> Hi all,
>
> Thanks to everyone who came to the multi-table transactions talk at
> Lakehouse Day in Glasgow, and for the conversations afterwards.
> Several of you suggested continuing here. I'm new to the Polaris
> community: I built this as a proof of concept on Polaris, and this is
> the first time I'm bringing it to the list.
>
> Iceberg already has most of the foundation for transactions engines
> can use: the REST commitTransaction bundle commits several tables
> atomically, and table branches can stage writes. What's missing is the
> engine usable transaction around them: BEGIN ... COMMIT across
> statements, with one consistent read point across tables. The
> discussion of this on the Iceberg list has been quiet for a while, and
> I built the PoC to restart it with something concrete.
>
> The PoC runs Spark and Trino against a Polaris fork: multi-statement
> transactions at READ COMMITTED and REPEATABLE READ, a read point
> shared across engines, and fan-out writes from independent workers
> that commit atomically. After conversations with Iceberg folks at the
> conference, I'm moving the write path onto Iceberg table branches, so
> writes become standard REST updates; I'll follow up in this thread
> with the updated code.
>
> For example, in Spark with the Polaris plugin:
>
>     BEGIN TRANSACTION;
>     INSERT INTO customers VALUES (22, 'joe', 'joe@example.com');
>     INSERT INTO orders    VALUES (2002, 22, 9.99, 'pending');
>     COMMIT;   -- both tables become visible together, or neither does
>
> Readers in Spark or Trino, at either isolation level, never see the
> order without its customer.
>
> The change to the Iceberg REST surface is small: one optional
> loadTable parameter (the read point). Everything else is Polaris-only
> endpoints (minting a read point, fan-out coordination). So it could
> live in Polaris first as an opt-in extension, with an Iceberg proposal
> later if the community wants one.
>
> Would Polaris be interested in carrying it as an extension first? And
> since I'm new here, I'd appreciate guidance on the usual path for a
> change of this size: for example a design review here first, an issue
> to track it, time at a community sync, or going straight to small PRs.
> I'm happy to work whichever way suits the project. It's an early PoC
> with known gaps (listed in the design doc), and JDBC/PostgreSQL only
> for now.
>
> Talk, slides, demo video, design doc and code:
> https://github.com/qlong/lakehouse-transactions
>
> Thanks,
> Qiegang