[DISCUSS] Support for staged creates in multi-table transactions (commitTransaction)

[Ayush Saxena <ay...@gmail.com>]

Hi Folks,
I recently opened a PR to introduce support for staged table creations
within the commitTransaction REST API endpoint [1]. Because this
touches core transactional boundaries, I wanted to bring it to the dev
list for discussion and feedback.

Currently, Polaris throws a BadRequestException if a
CommitTransactionRequest contains an updateForStagedCreate (identified
by the AssertTableDoesNotExist requirement). This prevents compute
engines from executing multi-table transactions that involve creating
new tables alongside updating existing ones. From the Iceberg client's
perspective, it builds the table metadata locally and expects to
commit the new tables and updates in a single server roundtrip.

This PR aims to remove this restriction so Polaris can process mixed
batches of staged creates and regular updates atomically.

I'd love to get the community's thoughts on supporting this flow and
hear any high-level feedback you might have. The PR is here [1]

-Ayush

[1] https://github.com/apache/polaris/pull/4939

Re: [DISCUSS] Support for staged creates in multi-table transactions (commitTransaction)

[Yufei Gu <fl...@gmail.com>]

> From the Iceberg client's
> perspective, it builds the table metadata locally and expects to
> commit the new tables and updates in a single server roundtrip.

Ayush, why do we need a stage creation REST call if the engines commit both
in a single server roundtrip? Can you share more details about the use
case? One related question: if the engine skips the stage-create request,
how does it obtain the credentials needed to write the table data and
metadata?

That said, I think it is reasonable not to reject an updateForStagedCreate
in a multi-table transaction. My understanding is that the
updateForStagedCreate
REST call validates constraints and persists the table independently.
Whether or not the stage-create REST call occurred, the catalog should be
able to process a complete and valid updateForStagedCreate. In that case,
omitting any stage-create call may also be acceptable during
a commitTransaction. It doesn't affect the persistence at all.


Yufei


On Thu, Jul 9, 2026 at 7:34 PM Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi Ayush,
>
> Sorry about the delay. I'll try to review soon :)
>
> Cheers,
> Dmitri.
>
> On Wed, Jul 1, 2026 at 1:28 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> > Hi Folks,
> > I recently opened a PR to introduce support for staged table creations
> > within the commitTransaction REST API endpoint [1]. Because this
> > touches core transactional boundaries, I wanted to bring it to the dev
> > list for discussion and feedback.
> >
> > Currently, Polaris throws a BadRequestException if a
> > CommitTransactionRequest contains an updateForStagedCreate (identified
> > by the AssertTableDoesNotExist requirement). This prevents compute
> > engines from executing multi-table transactions that involve creating
> > new tables alongside updating existing ones. From the Iceberg client's
> > perspective, it builds the table metadata locally and expects to
> > commit the new tables and updates in a single server roundtrip.
> >
> > This PR aims to remove this restriction so Polaris can process mixed
> > batches of staged creates and regular updates atomically.
> >
> > I'd love to get the community's thoughts on supporting this flow and
> > hear any high-level feedback you might have. The PR is here [1]
> >
> > -Ayush
> >
> > [1] https://github.com/apache/polaris/pull/4939
> >
>

Re: [DISCUSS] Support for staged creates in multi-table transactions (commitTransaction)

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Ayush,

Sorry about the delay. I'll try to review soon :)

Cheers,
Dmitri.

On Wed, Jul 1, 2026 at 1:28 PM Ayush Saxena <ay...@gmail.com> wrote:

> Hi Folks,
> I recently opened a PR to introduce support for staged table creations
> within the commitTransaction REST API endpoint [1]. Because this
> touches core transactional boundaries, I wanted to bring it to the dev
> list for discussion and feedback.
>
> Currently, Polaris throws a BadRequestException if a
> CommitTransactionRequest contains an updateForStagedCreate (identified
> by the AssertTableDoesNotExist requirement). This prevents compute
> engines from executing multi-table transactions that involve creating
> new tables alongside updating existing ones. From the Iceberg client's
> perspective, it builds the table metadata locally and expects to
> commit the new tables and updates in a single server roundtrip.
>
> This PR aims to remove this restriction so Polaris can process mixed
> batches of staged creates and regular updates atomically.
>
> I'd love to get the community's thoughts on supporting this flow and
> hear any high-level feedback you might have. The PR is here [1]
>
> -Ayush
>
> [1] https://github.com/apache/polaris/pull/4939
>