[DISCUSS] Credential Refresh for Long Staged Create Operations

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

As Sung highlighted in today's Comminity Sync call, there is a general
problem with refreshing credentials vended in the context of a Staged
Create operation (usually CTAS).

The same problem is beling discussed in the Iceberg community [1]

Recap:

* It is generally not possible for a client to refresh credentials received
from the initial Staged Create REST API call because the related table is
not created until commit, thus leading to 404 response from the regular
/credentials endpoint

* Conceptually, AuthZ checks required on refreshing these credentials are
different between the initial request and the refresh.

The initial request implies that no data files exist yet and as such
resembles an ordinary CREATE TABLE request.

When the refresh credential call comes in, some data files are likely to
exist already. Therefore, the credential refresh request is more similar to
an UPDATE from the AuthZ perspective.

Optoins:

* Wait for the Iceberg REST API spec change.

This will allow associating a particular Stage Create "session" with a
particular client.

Yet, this does not address the AuthZ aspect at all.

* Pre-register a Table Entity at Staged Create time.

This will reserve the entity name for the anticipated commit and allow the
Polaris Authorizer to distinguish subsequent requests for refreshing
credentials from the initial "create" request.

* Use a different credential refresh URI for Staged Create environments.

This URI is part of the Staged Create REST response, IIRC, and can be
expected to override other client-side configs. The URI can use custom path
segments or query parameters to identity the specific Staged env. context.

* Something else? Please share ideas.

[1] https://lists.apache.org/thread/35lj7vhtqtwl5nv9rzpln4mw4fbh7gdp

Thanks,
Dmitri.

Re: [DISCUSS] Credential Refresh for Long Staged Create Operations

[Yufei Gu <fl...@gmail.com>]

One thing worth noting is that the Iceberg discussion in PR #15280(
https://github.com/apache/iceberg/pull/15280) and the dev@ thread
discussion[1] seems to have evolved away from the earlier stagingSession
idea and toward a stateless storage refresh token model. Server-side
sessions/state were discarded due to the complexity and the desire to
support stateless catalog implementations. To me, the storage refresh token
also feels better since it ties to the storage credential itself rather
than a specific staged create workflow. It avoids maintaining server-side
session and state and easier to extend to other refreshing use cases in the
future.

If we want to align with the Iceberg direction, the storage refresh token
should be closer to that long-term vision. We may table a staged-create
specific session or lease model now.

1. https://lists.apache.org/thread/35lj7vhtqtwl5nv9rzpln4mw4fbh7gdp

Yufei


On Fri, May 29, 2026 at 9:01 AM Robert Stupp <sn...@snazy.de> wrote:

> Hi all,
>
> One thing to add: the “different refresh URI” option is usable today,
> but mostly for Iceberg Java clients.
>
> Iceberg/Java FileIO implementations for S3, GCS and ADLS already have
> provider-specific refresh endpoint config properties.
> So Polaris could return a URI specific to each staged-create in the
> LoadTableResult and that endpoint could vend fresh credentials for the
> staged location/context.
>
> But that URI cannot be the proof by itself.
> It either needs to contain a signed, short-lived staged operation token, or
> reference server-side staged/lease state that Polaris validates.
> This requires either cryptographic infrastructure or persisted state.
>
> It's probably worth pointing out: this is not a portable Iceberg REST
> solution.
> The Iceberg spec work was apache/iceberg#15280 [1], moving toward a
> StorageCredential refresh-token model.
> That PR was closed stale, not rejected, so there is no standard
> cross-client contract yet.
>
> So maybe the practical path might be:
> Use the Java hook for now, but model the Polaris side as a staged
> operation/lease with cryptographic or server-side proof, so it can map
> cleanly to the Iceberg refresh-token design if/when that comes back.
>
> Robert
>
> [1] https://github.com/apache/iceberg/pull/15280
>
>
> On Fri, May 29, 2026 at 1:20 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > As Sung highlighted in today's Comminity Sync call, there is a general
> > problem with refreshing credentials vended in the context of a Staged
> > Create operation (usually CTAS).
> >
> > The same problem is beling discussed in the Iceberg community [1]
> >
> > Recap:
> >
> > * It is generally not possible for a client to refresh credentials
> received
> > from the initial Staged Create REST API call because the related table is
> > not created until commit, thus leading to 404 response from the regular
> > /credentials endpoint
> >
> > * Conceptually, AuthZ checks required on refreshing these credentials are
> > different between the initial request and the refresh.
> >
> > The initial request implies that no data files exist yet and as such
> > resembles an ordinary CREATE TABLE request.
> >
> > When the refresh credential call comes in, some data files are likely to
> > exist already. Therefore, the credential refresh request is more similar
> to
> > an UPDATE from the AuthZ perspective.
> >
> > Optoins:
> >
> > * Wait for the Iceberg REST API spec change.
> >
> > This will allow associating a particular Stage Create "session" with a
> > particular client.
> >
> > Yet, this does not address the AuthZ aspect at all.
> >
> > * Pre-register a Table Entity at Staged Create time.
> >
> > This will reserve the entity name for the anticipated commit and allow
> the
> > Polaris Authorizer to distinguish subsequent requests for refreshing
> > credentials from the initial "create" request.
> >
> > * Use a different credential refresh URI for Staged Create environments.
> >
> > This URI is part of the Staged Create REST response, IIRC, and can be
> > expected to override other client-side configs. The URI can use custom
> path
> > segments or query parameters to identity the specific Staged env.
> context.
> >
> > * Something else? Please share ideas.
> >
> > [1] https://lists.apache.org/thread/35lj7vhtqtwl5nv9rzpln4mw4fbh7gdp
> >
> > Thanks,
> > Dmitri.
> >
>

Re: [DISCUSS] Credential Refresh for Long Staged Create Operations

[Robert Stupp <sn...@snazy.de>]

Hi all,

One thing to add: the “different refresh URI” option is usable today,
but mostly for Iceberg Java clients.

Iceberg/Java FileIO implementations for S3, GCS and ADLS already have
provider-specific refresh endpoint config properties.
So Polaris could return a URI specific to each staged-create in the
LoadTableResult and that endpoint could vend fresh credentials for the
staged location/context.

But that URI cannot be the proof by itself.
It either needs to contain a signed, short-lived staged operation token, or
reference server-side staged/lease state that Polaris validates.
This requires either cryptographic infrastructure or persisted state.

It's probably worth pointing out: this is not a portable Iceberg REST
solution.
The Iceberg spec work was apache/iceberg#15280 [1], moving toward a
StorageCredential refresh-token model.
That PR was closed stale, not rejected, so there is no standard
cross-client contract yet.

So maybe the practical path might be:
Use the Java hook for now, but model the Polaris side as a staged
operation/lease with cryptographic or server-side proof, so it can map
cleanly to the Iceberg refresh-token design if/when that comes back.

Robert

[1] https://github.com/apache/iceberg/pull/15280


On Fri, May 29, 2026 at 1:20 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> As Sung highlighted in today's Comminity Sync call, there is a general
> problem with refreshing credentials vended in the context of a Staged
> Create operation (usually CTAS).
>
> The same problem is beling discussed in the Iceberg community [1]
>
> Recap:
>
> * It is generally not possible for a client to refresh credentials received
> from the initial Staged Create REST API call because the related table is
> not created until commit, thus leading to 404 response from the regular
> /credentials endpoint
>
> * Conceptually, AuthZ checks required on refreshing these credentials are
> different between the initial request and the refresh.
>
> The initial request implies that no data files exist yet and as such
> resembles an ordinary CREATE TABLE request.
>
> When the refresh credential call comes in, some data files are likely to
> exist already. Therefore, the credential refresh request is more similar to
> an UPDATE from the AuthZ perspective.
>
> Optoins:
>
> * Wait for the Iceberg REST API spec change.
>
> This will allow associating a particular Stage Create "session" with a
> particular client.
>
> Yet, this does not address the AuthZ aspect at all.
>
> * Pre-register a Table Entity at Staged Create time.
>
> This will reserve the entity name for the anticipated commit and allow the
> Polaris Authorizer to distinguish subsequent requests for refreshing
> credentials from the initial "create" request.
>
> * Use a different credential refresh URI for Staged Create environments.
>
> This URI is part of the Staged Create REST response, IIRC, and can be
> expected to override other client-side configs. The URI can use custom path
> segments or query parameters to identity the specific Staged env. context.
>
> * Something else? Please share ideas.
>
> [1] https://lists.apache.org/thread/35lj7vhtqtwl5nv9rzpln4mw4fbh7gdp
>
> Thanks,
> Dmitri.
>