Re: [DISCUSS] Cloudflare R2 support with scoped credential vending

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

Recap from today's community sync call:

* Actual set of values acceptable for "credentialVendingMechanism" is a
finite list. Initially "STS" and "R2" will be supported by Polaris (exact
names TBD during R2 implementation).

* Servers validate values in "credentialVendingMechanism" in runtime after
parsing the request.

* The Open API doc defines "credentialVendingMechanism" as a string to
allow for flexibility in auto-generated code. Clients / parsers do not have
to re-build when a server adds a new option for
"credentialVendingMechanism".

Please correct me if I missed anything.

Does this sound acceptable?

Thanks,
Dmitri.

On Wed, Sep 30, 2026 at 1:30 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi Yufei,
>
> Thanks for providing the details! Replying inline.
>
> > When a client creates a catalog with credentialVendingMechanism: "m1",
> > what is the expected behavior?
>
> This depends on the meaning of "m1" and the specific server build.
>
> In Apache Polaris "m1" is not meaningful and is not supported.
> The server will reject this value on the corresponding catalog
> update request. This is handled in PolarisAdminService in the current PR
> state.
>
> > 1.  The server supports it: This is the ideal scenario (happy path).
>
> Indeed.
>
> > 2.  The server does not support it: This creates inconsistency if a
> client
> > that works with Server A fails on Server B.
>
> In my view this is not an inconsistency as different servers do not have
> to provide the same capabilities.
>
> Creating a catalog with a specific credentialVendingMechanism is an
> administrative task. The admin user must be aware of what the server
> supports. This is a deployment / configuration concern. Not a runtime
> compatibility concern.
>
> > 3.  The server supports "m1" differently: This is the worst-case
> scenario,
> > where the client's expectations conflict with the server's actual
> behavior.
>
> This is a good point!
>
> My take is that if "m1" is defined in Apache Polaris, then all server
> builds must implement it the same way.
>
> The definition will be expressed by means of S3CredentialVendingMechanism
> classes included in standard Apache Polaris releases (and hopefully docs
> too).
>
> Currently only "STS" and "DEFAULT" (falling back on STS) are defined now.
>
> If "m1" is not defined in Apache Polaris, custom builds are free to assign
> any meaning to it and do not have to be mutually compatible. This is the
> main point for extensibility.
>
> Note that custom builds are not "Apache Polaris" anymore and cannot
> claim to be the "same" as Apache Polaris. This is a matter of ASF
> trademarks.
> Normally, custom builds say something like "Powered by Apache Polaris".
>
> Again, the admin user setting "m1" as a vending mechanism in a particular
> server must be aware of what that server provides and understand
> the meaning of "m1" in that context. The onus to describe the meaning of
> "m1" to users in this case rests with the custom server developers.
>
> We can talk about preventing name overlaps, but I do not think establishing
> any kind of name registry is worth the trouble. In this day and age, I
> believe
> all downstream project developers are aware of name overlap issues.
> and will follow normal practices to avoid name collisions.
>
> I hope this clears your concerns.
>
> Cheers,
> Dmitri.
>
> On Wed, Sep 30, 2026 at 1:02 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi Dmitri,
>>
>> When a client creates a catalog with credentialVendingMechanism: "m1",
>> what
>> is the expected behavior?
>>
>> Here is a suitaion if we allow a freeform string, the client might
>> encounter three different scenarios depending on the server:
>> 1.  The server supports it: This is the ideal scenario (happy path).
>> 2.  The server does not support it: This creates inconsistency if a client
>> that works with Server A fails on Server B.
>> 3.  The server supports "m1" differently: This is the worst-case scenario,
>> where the client's expectations conflict with the server's actual
>> behavior.
>>
>> I don't think this is the situation we want users to run into. The fix is
>> quite simple, using a specified values instead of a freefrom string.
>>
>> Yufei
>>
>>
>> On Tue, Sep 29, 2026 at 9:48 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>> > Hi all,
>> >
>> > Since concerns were raised, it's important to take a moment to align
>> > rather than rushing. That's the way we drive consensus :)
>> >
>> > On the technical question of string vs enum for
>> > credentialVendingMechanism, here is my take:
>> >
>> > 1. This field is part of the Management API for catalog
>> > administrators. Iceberg REST clients (Spark, Trino, ...) never see or
>> > interact with credentialVendingMechanism. They only consume standard
>> > vended credentials (s3.* session tokens/keys) during table
>> > load/commit. There is no risk of breaking query engine clients across
>> > deployments.
>> > 2. One of our primary goals when discussing S3-compatible stores (R2,
>> > MinIO, ...) was to keep credential vending pluggable via SPI rather
>> > than hardcoding vendor-specific paths in core. If we use a closed enum
>> > in the OpenAPI spec, it means:
>> >   * any new mechanism (including R2 in Phase 2 or MinIO) requires a spec
>> > change.
>> >   * any downstream deployment or plugin providing a custom mechanism
>> > would break OpenAPI-generated client SDKs due to strict enum
>> > validation.
>> >   * runtime validation on the server (returning HTTP 400 with a clear
>> > message) combined with the realm allowlist provides the right
>> > server-side guardrail without restricting extensibility.
>> > 3. Yufei's concerns about a clear contract and avoiding confusion is
>> > valid. We can address contract clarity without closing the enum:
>> >   * we keep type: string, but document the well-known/standard values
>> > (STS, DEFAULT) explicitly in the description and examples in the
>> > OpenAPI spec (as PR #5513 does).
>> >   * we ensure the server validates cleanly with case-insensitivity (or
>> > canonical normalization) so administrators get immediate, actionable
>> > feedback on typos.
>> >
>> > PR #5513 does a solid job laying this foundation so Austen can move
>> > forward with Phase 2. If we keep it as a string with well-documented
>> > standard values and server-siide validation, I think it's the right
>> > balance between spec clarify and pluggability.
>> >
>> > Thoughts?
>> >
>> > Regards
>> > JB
>> >
>> > On Wed, Sep 30, 2026 at 12:50 AM Dmitri Bourlatchkov <di...@apache.org>
>> > wrote:
>> > >
>> > > Hi Yufei,
>> > >
>> > > > > Conversely, an enum will require defining all possible vending
>> > > mechanisms in
>> > > > > the Polaris API spec.
>> > >
>> > > > Yes. Otherwise, a client may failed in certain deployments.
>> > >
>> > > Could you describe this failure mode in terms of API request/response
>> > flow?
>> > >
>> > > Thanks,
>> > > Dmitri.
>> > >
>> > > On Tue, Sep 29, 2026 at 2:06 PM Yufei Gu <fl...@gmail.com>
>> wrote:
>> > >
>> > > > > Conversely, an enum will require defining all possible vending
>> > > > mechanisms in
>> > > > the Polaris API spec.
>> > > >
>> > > > Yes. Otherwise, a client may failed in certain deployments.
>> > > >
>> > > > > This will prevent downstream projects from using custom vending
>> > > > mechanisms.
>> > > >
>> > > > No. if downstream needs custom vending mechanisms, they can always
>> > build
>> > > > their own Polaris.
>> > > >
>> > > > Yufei
>> > > >
>> > > >
>> > > > On Tue, Sep 29, 2026 at 11:01 AM Dmitri Bourlatchkov <
>> dimas@apache.org
>> > >
>> > > > wrote:
>> > > >
>> > > > > Hi Yufei,
>> > > > >
>> > > > > > Using an enum will clarify the REST spec contract.
>> > > > >
>> > > > > I'm not sure I understand your point. Current PR clearly describes
>> > the
>> > > > > meaning of "credentialVendingMechanism" and lists options
>> available
>> > > > > out-of-the-box.
>> > > > >
>> > > > > What does an enum achieve, that (current) runtime validation does
>> > not?
>> > > > >
>> > > > > Conversely, an enum will require defining all possible vending
>> > mechanisms
>> > > > > in the Polaris API spec. This will prevent downstream projects
>> from
>> > using
>> > > > > custom vending mechanisms.
>> > > > >
>> > > > > I believe Polaris as a project currently promotes downstream
>> > > > > customizability.
>> > > > >
>> > > > > Cheers,
>> > > > > Dmitri.
>> > > > >
>> > > > > On Tue, Sep 29, 2026 at 1:21 PM Yufei Gu <fl...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > > I don't think we have consensus on the PR, esp. a client should
>> NOT
>> > > > > depend
>> > > > > > on a specific deployment. Using an enum will clarify the REST
>> spec
>> > > > > > contract.
>> > > > > >
>> > > > > > Yufei
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Sep 29, 2026 at 7:46 AM Dmitri Bourlatchkov <
>> > dimas@apache.org>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi All,
>> > > > > > >
>> > > > > > > PR [5513] has been in review for quite some time. I believe we
>> > have
>> > > > > > general
>> > > > > > > consensus on the Management API changes and java code changes.
>> > > > > > >
>> > > > > > > The only point that may still have divergent opinions is
>> whether
>> > to
>> > > > use
>> > > > > > > free text or enum for "credentialVendingMechanism". I
>> personally
>> > > > > support
>> > > > > > > free text (rationale explained in previous comments). This
>> > > > conversation
>> > > > > > has
>> > > > > > > been dormant for a few days, so I assume lazy consensus on the
>> > > > current
>> > > > > > > state of the PR (free text). In any case switching from free
>> > text to
>> > > > > enum
>> > > > > > > is a non-breaking API change before the next release in case
>> the
>> > > > > > consensus
>> > > > > > > shifts and we need to adjust. We have about one month for
>> > followup
>> > > > API
>> > > > > > > changes.
>> > > > > > >
>> > > > > > > The java code introduced in this PR is not rigid and can
>> change
>> > at
>> > > > any
>> > > > > > time
>> > > > > > > per our evolution guidelines [1].
>> > > > > > >
>> > > > > > > I propose merging PR [5513] on Sept 30 unless fresh blocking
>> > concerns
>> > > > > are
>> > > > > > > raised.
>> > > > > > >
>> > > > > > > [1] https://polaris.apache.org/releases/1.8.0/evolution/
>> > > > > > >
>> > > > > > > [5513] https://github.com/apache/polaris/pull/5513
>> > > > > > >
>> > > > > > > Cheers,
>> > > > > > > Dmitri.
>> > > > > > >
>> > > > > > > On Tue, Sep 29, 2026 at 9:30 AM Austen Tomek <
>> > > > > > > Austen.Tomek@chicagotrading.com> wrote:
>> > > > > > >
>> > > > > > > > Hey all,
>> > > > > > > >
>> > > > > > > > Any chance I can get further reviews this week on the PR?
>> I’d
>> > like
>> > > > to
>> > > > > > get
>> > > > > > > > moving onto Phase 2 here if there are no further issues to
>> > address.
>> > > > > > > >
>> > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac>
>> > > > > > > >
>> > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
>> > > > > > > > Date: Thursday, September 24, 2026 at 17:33
>> > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped
>> > credential
>> > > > > > > vending
>> > > > > > > >
>> > > > > > > > Hi Yufei,
>> > > > > > > >
>> > > > > > > > > The REST spec is a contract between Polaris servers and
>> > clients
>> > > > > [...]
>> > > > > > > >
>> > > > > > > > I tend to think that this statement is a bit too strong. The
>> > > > > Management
>> > > > > > > API
>> > > > > > > > spec defines a protocol for clients and servers to
>> communicate.
>> > > > > > > >
>> > > > > > > > > [...] This would help avoid a situation where a Polaris
>> > client
>> > > > > works
>> > > > > > > > > only with certain deployments.
>> > > > > > > >
>> > > > > > > > Since Polaris has been developed as a customizable system,
>> a
>> > REST
>> > > > > API
>> > > > > > > > specification defined by Apache Polaris cannot control the
>> > > > behaviour
>> > > > > of
>> > > > > > > all
>> > > > > > > > possible downstream environments. The purpose of the spec
>> is to
>> > > > > ensure
>> > > > > > > > protocol compatibility.
>> > > > > > > >
>> > > > > > > > Note that the value of `credentialVendingMechanism` has no
>> > bearing
>> > > > on
>> > > > > > > > any logic clients need to execute. It is an opaque parameter
>> > > > > controlled
>> > > > > > > > by the Polaris administrator.
>> > > > > > > >
>> > > > > > > > > The new credentialVendingMechanism field in the REST spec
>> > looks
>> > > > > > > > > reasonable. My only suggestion is to use an enum instead
>> of a
>> > > > > string.
>> > > > > > > > [...]
>> > > > > > > >
>> > > > > > > > The above points are exactly the reason why the Management
>> API
>> > > > cannot
>> > > > > > > > enumerate all possible vending mechanisms.
>> > > > > > > >
>> > > > > > > > The same applies to storage types too. However, we do not
>> have
>> > any
>> > > > > > > > practical
>> > > > > > > > examples of 3rd party storage systems that go beyond
>> > S3/GCS/ADLS
>> > > > > (yet).
>> > > > > > > >
>> > > > > > > > Cheers,
>> > > > > > > > Dmitri.
>> > > > > > > >
>> > > > > > > > On Thu, Sep 24, 2026 at 4:38 PM Yufei Gu <
>> flyrain000@gmail.com
>> > >
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Thanks Austen. It's fair that endpoints are not reliable
>> > given
>> > > > that
>> > > > > > > some
>> > > > > > > > > s3-compatible can use arbitrary hostnames.
>> > > > > > > > > I also think stsUnavailable is fine to keep as-is. I don’t
>> > think
>> > > > we
>> > > > > > > need
>> > > > > > > > to
>> > > > > > > > > change its scope or repurpose it.
>> > > > > > > > >
>> > > > > > > > > The new credentialVendingMechanism field in the REST spec
>> > looks
>> > > > > > > > reasonable.
>> > > > > > > > > My only suggestion is to use an enum instead of a string.
>> The
>> > > > REST
>> > > > > > spec
>> > > > > > > > is
>> > > > > > > > > a contract between Polaris servers and clients, so I’d
>> > expect it
>> > > > to
>> > > > > > > > define
>> > > > > > > > > the valid values and their meanings rather than leaving
>> them
>> > up
>> > > > to
>> > > > > > each
>> > > > > > > > > deployment. This would help avoid a situation where a
>> Polaris
>> > > > > client
>> > > > > > > > works
>> > > > > > > > > only with certain deployments. Deployment-specific
>> mechanisms
>> > > > > > shouldn’t
>> > > > > > > > be
>> > > > > > > > > part of the standard REST contract; users who need a
>> custom
>> > > > > mechanism
>> > > > > > > can
>> > > > > > > > > always extend the server in their own deployment.
>> > > > > > > > >
>> > > > > > > > > > Either way the operator needs a per-realm switch like
>> > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control
>> which
>> > realm
>> > > > > can
>> > > > > > > > mint
>> > > > > > > > > R2 credentials from the operators Cloudflare account.
>> > > > > > > > >
>> > > > > > > > > Could you clarify why we need a separate per-realm switch?
>> > Once
>> > > > the
>> > > > > > > > storage
>> > > > > > > > > configuration is set at the catalog level to select a
>> > specific
>> > > > > > > mechanism,
>> > > > > > > > > the catalog will have no choice but to support it. Do we
>> need
>> > > > > another
>> > > > > > > > layer
>> > > > > > > > > of gating?
>> > > > > > > > >
>> > > > > > > > > Yufei
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Wed, Sep 23, 2026 at 5:53 AM Austen Tomek <
>> > > > > > > > > Austen.Tomek@chicagotrading.com> wrote:
>> > > > > > > > >
>> > > > > > > > > > Hey Yufei,
>> > > > > > > > > >
>> > > > > > > > > > Apologies for missing your question.
>> > > > > > > > > >
>> > > > > > > > > > I don’t think Endpoint is sufficient.
>> > > > > > > > > >
>> > > > > > > > > > Endpoint matching only works where the hostname
>> identifies
>> > the
>> > > > > > vendor
>> > > > > > > > > like
>> > > > > > > > > > R2’s public hostnames that are recognizable, MinIO and
>> > other
>> > > > > > > > self-hosted
>> > > > > > > > > > S3-compatible stores use arbitrary hostnames so there
>> > would be
>> > > > > > > nothing
>> > > > > > > > to
>> > > > > > > > > > match on. On top of that, if a match fails it would just
>> > > > silently
>> > > > > > > fall
>> > > > > > > > > back
>> > > > > > > > > > to STS.
>> > > > > > > > > >
>> > > > > > > > > > Right now the endpoint just says how you’ll connect not
>> > how the
>> > > > > > > > > > credentials are vended. There's endpoint and also
>> > > > > endpointInternal
>> > > > > > > for
>> > > > > > > > > > server-side access, so it would be unclear which of the
>> two
>> > > > > > decides.
>> > > > > > > > > >
>> > > > > > > > > > For your other question, there is no second mechanism
>> for
>> > R2
>> > > > > today,
>> > > > > > > but
>> > > > > > > > > > for other self-hosted stores the same endpoint could be
>> > served
>> > > > by
>> > > > > > STS
>> > > > > > > > or
>> > > > > > > > > by
>> > > > > > > > > > a different token mechanism chosen per catalog.
>> > > > > > > > > >
>> > > > > > > > > > stsUnavailable keeps it’s current behavior in the PR
>> (that
>> > was
>> > > > > > > > Prithvi’s
>> > > > > > > > > > point earlier in the thread)
>> > > > > > > > > >
>> > > > > > > > > > Either way the operator needs a per-realm switch like
>> > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control
>> which
>> > > > realm
>> > > > > > can
>> > > > > > > > > mint
>> > > > > > > > > > R2 credentials from the operators Cloudflare account.
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Austen
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac>
>> > > > > > > > > >
>> > > > > > > > > > From: Yufei Gu <fl...@gmail.com>
>> > > > > > > > > > Date: Monday, September 21, 2026 at 17:53
>> > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped
>> > > > > credential
>> > > > > > > > > vending
>> > > > > > > > > >
>> > > > > > > > > > Hi Austen,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for the update. After looking at PR #5513, I
>> think
>> > we
>> > > > > should
>> > > > > > > > first
>> > > > > > > > > > clarify whether we need a new credentialVendingMechanism
>> > field
>> > > > in
>> > > > > > the
>> > > > > > > > > > spec/config.
>> > > > > > > > > >
>> > > > > > > > > > Austen, you mentioned that the R2 account ID and
>> > jurisdiction
>> > > > > > already
>> > > > > > > > > come
>> > > > > > > > > > from the endpoint. Could we also use the existing field
>> > like
>> > > > > > > > `endpoint`,
>> > > > > > > > > to
>> > > > > > > > > > recognize R2 and select its credential-vending
>> mechanism,
>> > while
>> > > > > > > keeping
>> > > > > > > > > the
>> > > > > > > > > > current stsUnavailable behavior unchanged?
>> > > > > > > > > >
>> > > > > > > > > > Are there cases where the endpoint isn’t enough, or
>> where
>> > users
>> > > > > > need
>> > > > > > > to
>> > > > > > > > > > choose different vending mechanisms for the same
>> > endpoint(e.g.,
>> > > > > R2)
>> > > > > > > per
>> > > > > > > > > > catalog?
>> > > > > > > > > >
>> > > > > > > > > > On Wed, Sep 16, 2026 at 7:24 AM Austen Tomek <
>> > > > > > > > > > Austen.Tomek@chicagotrading.com> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Yufei, Dimitri,
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks, both. PR 1 is ready for review:
>> > > > > > > > > > > https://github.com/apache/polaris/pull/5513
>> > > > > > > > > > >
>> > > > > > > > > > > I pushed a revision last night that takes in Dimitri’s
>> > > > > comments.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >   *
>> > > > > > > > > > > I changed from issuer to credentialVendingMechanism.
>> It’s
>> > > > also
>> > > > > > now
>> > > > > > > a
>> > > > > > > > > > plain
>> > > > > > > > > > > string and the realm allowlist is
>> > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS, which
>> still
>> > > > > defaults
>> > > > > > to
>> > > > > > > > > [STS]
>> > > > > > > > > > >   *
>> > > > > > > > > > > Mechanisms are CDI beans found by their identifier, as
>> > > > Dimitri
>> > > > > > > > > proposed.
>> > > > > > > > > > >   *
>> > > > > > > > > > > The CLOUDFLARE_R2 bean and the R2 vending code will
>> > follow in
>> > > > > the
>> > > > > > > > next
>> > > > > > > > > PR
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > CI workflow is waiting approval to run.
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks,
>> > > > > > > > > > > Austen
>> > > > > > > > > > >
>> > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac>
>> > > > > > > > > > >
>> > > > > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
>> > > > > > > > > > > Date: Tuesday, September 15, 2026 at 20:00
>> > > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with
>> scoped
>> > > > > > credential
>> > > > > > > > > > vending
>> > > > > > > > > > >
>> > > > > > > > > > > Hi All,
>> > > > > > > > > > >
>> > > > > > > > > > > > Having getCloudflareR2Endpoint on
>> > > > AwsStorageConfigurationInfo
>> > > > > > > > > > >
>> > > > > > > > > > > Yes, this does feel awkward.
>> > > > > > > > > > >
>> > > > > > > > > > > From my POV pursuing the option with the
>> > "credentialIssuers"
>> > > > > > > property
>> > > > > > > > > > looks
>> > > > > > > > > > > promising. This property is fairly generic and fits
>> > naturally
>> > > > > > into
>> > > > > > > > the
>> > > > > > > > > > > config, given that we already target multiple
>> backends.
>> > > > > > > > > > >
>> > > > > > > > > > > Cheers,
>> > > > > > > > > > > Dmitri.
>> > > > > > > > > > >
>> > > > > > > > > > > On Tue, Sep 15, 2026 at 8:48 PM Yufei Gu <
>> > > > flyrain000@gmail.com
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi Austen, Dmitri,
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Having getCloudflareR2Endpoint on
>> > > > > AwsStorageConfigurationInfo
>> > > > > > > > > > > >
>> > > > > > > > > > > > That does feel awkward. However,
>> > > > AwsStorageConfigurationInfo
>> > > > > > has
>> > > > > > > > > > > supported
>> > > > > > > > > > > > S3-compatible backends beyond AWS for a while, so
>> > there is
>> > > > > > > > precedent
>> > > > > > > > > > > (e.g.,
>> > > > > > > > > > > > MinIO, etc.). The underlying mismatch is that we
>> call
>> > the
>> > > > > > storage
>> > > > > > > > > type
>> > > > > > > > > > > S3,
>> > > > > > > > > > > > but its configuration mixes shared S3 settings with
>> > > > > > AWS-specific
>> > > > > > > > > ones:
>> > > > > > > > > > > >
>> > > > > > > > > > > > AwsStorageConfigurationInfo → storage type: S3
>> > > > > > > > > > > >   ├── Common S3 settings
>> > > > > > > > > > > >   │     endpoint, region, pathStyleAccess,
>> > allowedLocations
>> > > > > > > > > > > >   └── AWS-specific settings
>> > > > > > > > > > > >         roleArn, externalId, STS, KMS
>> > > > > > > > > > > >
>> > > > > > > > > > > > I’m open to separating the shared S3 settings from
>> > > > > > > > provider-specific
>> > > > > > > > > > > > configuration if that helps, but I don’t think we
>> need
>> > to
>> > > > > > settle
>> > > > > > > > on a
>> > > > > > > > > > > > broader refactor now.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Let me know when the PR is ready for review. I’d be
>> > happy
>> > > > to
>> > > > > > > take a
>> > > > > > > > > > > closer
>> > > > > > > > > > > > look.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yufei
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yufei
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:18 PM Dmitri Bourlatchkov
>> <
>> > > > > > > > > dimas@apache.org>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi Austen,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > +1 to adding "credentialIssues" or similar
>> (renamed)
>> > > > > property
>> > > > > > > to
>> > > > > > > > S3
>> > > > > > > > > > > > config.
>> > > > > > > > > > > > > In fact, I was looking exactly for something like
>> > that
>> > > > last
>> > > > > > > week
>> > > > > > > > > but
>> > > > > > > > > > > the
>> > > > > > > > > > > > > idea did not quite condense in my mind :)
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > I posted some preliminary comments on the draft
>> PR.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Just for community awareness, I'd like to propose
>> > here
>> > > > that
>> > > > > > we
>> > > > > > > > use
>> > > > > > > > > > CDI
>> > > > > > > > > > > > for
>> > > > > > > > > > > > > finding actual "issuer" / "mechanism"
>> > implementations.
>> > > > The
>> > > > > > > > codebase
>> > > > > > > > > > has
>> > > > > > > > > > > > > many examples of this. The closest match in this
>> > case is
>> > > > > > > > > > > > > probably PolarisStorageIntegrationProvider.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > This way the API does not have to enumerate all
>> > possible
>> > > > > > > values.
>> > > > > > > > > > > > Downstream
>> > > > > > > > > > > > > builds are free to add different implementations
>> > without
>> > > > > > having
>> > > > > > > > to
>> > > > > > > > > > > change
>> > > > > > > > > > > > > the API spec. The "allowed" list still makes
>> sense.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > In general, I like the direction this feature is
>> > taking,
>> > > > > > > > although I
>> > > > > > > > > > > > expect
>> > > > > > > > > > > > > PR reviews may take a few rounds :)
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Cheers,
>> > > > > > > > > > > > > Dmitri.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:38 PM Austen Tomek <
>> > > > > > > > > > > > > Austen.Tomek@chicagotrading.com> wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi all,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Closing the loop since my last note and
>> answering
>> > the
>> > > > > > > questions
>> > > > > > > > > > that
>> > > > > > > > > > > > have
>> > > > > > > > > > > > > > come up.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Dmitri's question on role ARNs: R2 doesn’t have
>> > > > AWS-style
>> > > > > > IAM
>> > > > > > > > > roles
>> > > > > > > > > > > or
>> > > > > > > > > > > > > > ARNs. Our implementation signs temporary
>> > credentials
>> > > > > with a
>> > > > > > > > > parent
>> > > > > > > > > > > > > token’s
>> > > > > > > > > > > > > > secret, scoped to a bucket and a set of
>> prefixes.
>> > The
>> > > > > > account
>> > > > > > > > ID
>> > > > > > > > > > and
>> > > > > > > > > > > > > > jurisdiction come from the R2 endpoint. Putting
>> the
>> > > > > account
>> > > > > > > ID
>> > > > > > > > in
>> > > > > > > > > > > > roleArn
>> > > > > > > > > > > > > > or userArn would give those fields a different
>> > meaning,
>> > > > > so
>> > > > > > > I’ve
>> > > > > > > > > > left
>> > > > > > > > > > > > them
>> > > > > > > > > > > > > > absent for CLOUDFLARE_R2.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I pursued Option B. The S3 configuration gains
>> one
>> > new
>> > > > > > field,
>> > > > > > > > > > > > > > credentialIssuer, with values STS
>> > > > > > > > > > > > > > (default) and CLOUDFLARE_R2 and a realm
>> allowlist
>> > > > > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_ISSUERS that defaults
>> > > > > > > > > > > > > > to [STS]. I left stsUnavailable untouched
>> > (Prithvi) so
>> > > > it
>> > > > > > > still
>> > > > > > > > > > means
>> > > > > > > > > > > > > > "vend nothing". The
>> > > > > > > > > > > > > > credentialIssuer is the explicit sign. The R2
>> > config
>> > > > > > rejects
>> > > > > > > > > > > > > > stsUnavailable along with a host of other STS
>> and
>> > KMS
>> > > > > > fields.
>> > > > > > > > I'm
>> > > > > > > > > > > > running
>> > > > > > > > > > > > > > this in our non-production environment at the
>> > moment.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Draft PR for stage 1 (config field, API,
>> allowlist,
>> > > > > gates):
>> > > > > > > > > > > > > > https://github.com/apache/polaris/pull/5513.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I still see the appeal of a separate R2
>> > configuration,
>> > > > as
>> > > > > > > > Dmitri
>> > > > > > > > > > > > > > suggested. My original reason for separating it
>> > was to
>> > > > > keep
>> > > > > > > the
>> > > > > > > > > > > > > R2-specific
>> > > > > > > > > > > > > > configuration and validation together. Having
>> > > > > > > > > > getCloudflareR2Endpoint
>> > > > > > > > > > > > on
>> > > > > > > > > > > > > > AwsStorageConfigurationInfo is one part I’d
>> still
>> > like
>> > > > > > > feedback
>> > > > > > > > > on.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I wanted to put a working version of Option B in
>> > front
>> > > > of
>> > > > > > > > > everyone
>> > > > > > > > > > so
>> > > > > > > > > > > > we
>> > > > > > > > > > > > > > could discuss the actual changes. In my earlier
>> > > > > > > implementation,
>> > > > > > > > > the
>> > > > > > > > > > > > > > separate config also meant a separate storage
>> type
>> > > > > sharing
>> > > > > > > > > > S3FileIO.
>> > > > > > > > > > > > That
>> > > > > > > > > > > > > > was the tradeoff that led me to try Option B.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Do the diffs change anyone’s lean here? And are
>> the
>> > > > names
>> > > > > > > > > > > > > credentialIssuer
>> > > > > > > > > > > > > > / STS / CLOUDFLARE_R2 /
>> > SUPPORTED_S3_CREDENTIAL_ISSUERS
>> > > > > > > > > acceptable?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > Austen
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Get Outlook for Mac<
>> > https://aka.ms/GetOutlookForMac>
>> > > > > > > > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
>> > > > > > > > > > > > > > Date: Sunday, September 13, 2026 at 19:50
>> > > > > > > > > > > > > > To: dev@polaris.apache.org <
>> dev@polaris.apache.org
>> > >
>> > > > > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support
>> with
>> > > > scoped
>> > > > > > > > > credential
>> > > > > > > > > > > > > vending
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Building on top of Ayush's and JB's suggestion,
>> I
>> > agree
>> > > > > > that
>> > > > > > > > > > > > jurisdiction
>> > > > > > > > > > > > > > can live inside the existing endpoint config
>> field.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > That leaves only Account ID, so potentially we
>> > could
>> > > > > store
>> > > > > > it
>> > > > > > > > in
>> > > > > > > > > > user
>> > > > > > > > > > > > or
>> > > > > > > > > > > > > > role ARN. Note that userArn is currently unused
>> in
>> > OSS
>> > > > > > code.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > That should allow us to keep existing
>> > > > > > > > AwsStorageConfigurationInfo
>> > > > > > > > > > for
>> > > > > > > > > > > > all
>> > > > > > > > > > > > > > S3 storage backend, but will require special
>> > parsing of
>> > > > > > > > user/role
>> > > > > > > > > > ARN
>> > > > > > > > > > > > > field
>> > > > > > > > > > > > > > to distinguish R2 from the other systems.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I'm still leaning towards the separate
>> > > > > > > > > R2StorageConfigurationInfo,
>> > > > > > > > > > > but
>> > > > > > > > > > > > > I'd
>> > > > > > > > > > > > > > be ok with overloading user/role ARN too if
>> other
>> > > > people
>> > > > > > > prefer
>> > > > > > > > > > > that. I
>> > > > > > > > > > > > > > guess a key question in this context is whether
>> R2
>> > has
>> > > > > the
>> > > > > > > > > concept
>> > > > > > > > > > of
>> > > > > > > > > > > > > role
>> > > > > > > > > > > > > > ARN at all.... but TBH I have no idea about
>> that. I
>> > > > hope
>> > > > > > > Austen
>> > > > > > > > > > might
>> > > > > > > > > > > > be
>> > > > > > > > > > > > > > able to clarify that.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Cheers,
>> > > > > > > > > > > > > > Dmitri.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Sun, Sep 13, 2026 at 9:18 AM Prithvi S <
>> > > > > > > > > > > prithvisivasankar@gmail.com
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Hi Austen,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Welcome, and thanks for starting with a
>> > [DISCUSS].
>> > > > Best
>> > > > > > way
>> > > > > > > > to
>> > > > > > > > > > > make a
>> > > > > > > > > > > > > > first
>> > > > > > > > > > > > > > > contribution :)
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > I think the thread is talking about two
>> layers,
>> > and
>> > > > we
>> > > > > > can
>> > > > > > > > keep
>> > > > > > > > > > > both
>> > > > > > > > > > > > > > > answers.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On the client side this should stay S3: s3://
>> > > > > locations,
>> > > > > > > s3.*
>> > > > > > > > > > > > > properties,
>> > > > > > > > > > > > > > > S3FileIO. Your PyIceberg / DuckDB / Iceberg
>> Java
>> > > > > results
>> > > > > > > > > already
>> > > > > > > > > > > show
>> > > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > unmodified clients work, so a dedicated
>> R2FileIO
>> > does
>> > > > > not
>> > > > > > > > seem
>> > > > > > > > > > > worth
>> > > > > > > > > > > > > it.
>> > > > > > > > > > > > > > > On the server side I lean toward Dmitri: a
>> > distinct
>> > > > > > > > > > > > > > > R2StorageConfigurationInfo and storage
>> > integration.
>> > > > > > > Issuance
>> > > > > > > > is
>> > > > > > > > > > not
>> > > > > > > > > > > > > STS,
>> > > > > > > > > > > > > > > and folding Cloudflare fields into
>> > > > > > > > AwsStorageConfigurationInfo
>> > > > > > > > > > > mixes
>> > > > > > > > > > > > > > > unrelated concepts. The middle ground to avoid
>> > is a
>> > > > new
>> > > > > > > type
>> > > > > > > > > that
>> > > > > > > > > > > is
>> > > > > > > > > > > > > > still
>> > > > > > > > > > > > > > > AWS-shaped underneath.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > One concrete caution if the Option B prototype
>> > stays
>> > > > on
>> > > > > > the
>> > > > > > > > S3
>> > > > > > > > > > > > config:
>> > > > > > > > > > > > > > > please do not overload stsUnavailable. Today
>> that
>> > > > flag
>> > > > > > > skips
>> > > > > > > > > > > > AssumeRole
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > disables vending. R2 wants the opposite. vend,
>> > just
>> > > > not
>> > > > > > via
>> > > > > > > > > STS.
>> > > > > > > > > > so
>> > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > flag would change meaning for existing MinIO /
>> > NetApp
>> > > > > > > > catalogs.
>> > > > > > > > > > If
>> > > > > > > > > > > we
>> > > > > > > > > > > > > > stay
>> > > > > > > > > > > > > > > on S3 config, we need an explicit vending
>> > signal; a
>> > > > > > > dedicated
>> > > > > > > > > R2
>> > > > > > > > > > > type
>> > > > > > > > > > > > > > > avoids that trap.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On config surface, jurisdiction can live in
>> > endpoint
>> > > > > > > > > (<account>.
>> > > > > > > > > > > > > > > r2.cloudflarestorage.com vs
>> <account>.eu.r2...).
>> > > > > Account
>> > > > > > > ID
>> > > > > > > > is
>> > > > > > > > > > > still
>> > > > > > > > > > > > > > > needed
>> > > > > > > > > > > > > > > for the JWT sub, so keeping an explicit
>> > accountId and
>> > > > > > > > deriving
>> > > > > > > > > > > > audience
>> > > > > > > > > > > > > > > from the endpoint seems enough. Parent tokens
>> > should
>> > > > > stay
>> > > > > > > in
>> > > > > > > > > > server
>> > > > > > > > > > > > > > config
>> > > > > > > > > > > > > > > via storageName.
>> > > > > > > > > > > > > > > Local signing fits the vending model.
>> > Fail-closed on
>> > > > > > > > > cross-bucket
>> > > > > > > > > > > and
>> > > > > > > > > > > > > > mixed
>> > > > > > > > > > > > > > > read/write grants is the right call. Prefix
>> > claims
>> > > > > should
>> > > > > > > > > follow
>> > > > > > > > > > > the
>> > > > > > > > > > > > > same
>> > > > > > > > > > > > > > > path comparison Polaris already uses, so we do
>> > not
>> > > > > mint a
>> > > > > > > > > > > credential
>> > > > > > > > > > > > > > wider
>> > > > > > > > > > > > > > > than the grant.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > +1 to small PRs: FileIO discriminator (#4486)
>> > > > > separately
>> > > > > > if
>> > > > > > > > R2
>> > > > > > > > > > > needs
>> > > > > > > > > > > > > it,
>> > > > > > > > > > > > > > > then management API + Java config, then
>> vending,
>> > > > Python
>> > > > > > CLI
>> > > > > > > > on
>> > > > > > > > > > its
>> > > > > > > > > > > > own.
>> > > > > > > > > > > > > > > Sketching MinIO / Backblaze against the same
>> > fields
>> > > > is
>> > > > > a
>> > > > > > > > useful
>> > > > > > > > > > > > design
>> > > > > > > > > > > > > > > check; a generic non-STS SPI can wait for a
>> > second
>> > > > > > backend.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Regards,
>> > > > > > > > > > > > > > > Prithvi S
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Fri, Sep 11, 2026 at 4:31 AM Dmitri
>> > Bourlatchkov <
>> > > > > > > > > > > > dimas@apache.org>
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Hi Austen,
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Thanks for starting this contribution!
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > After briefly looking through your PR [1] I
>> > think
>> > > > > > > > > > > > > > > > adding R2StorageConfigInfo makes sense.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Even though "aws" in
>> > AwsStorageConfigurationInfo is
>> > > > > > > longer
>> > > > > > > > > > > implies
>> > > > > > > > > > > > > AWS
>> > > > > > > > > > > > > > > > services, and MinIO, Rust, Ozone are
>> supported
>> > just
>> > > > > as
>> > > > > > > > well,
>> > > > > > > > > R2
>> > > > > > > > > > > > > appears
>> > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > be significantly different from STS-based S3
>> > > > systems
>> > > > > > > that a
>> > > > > > > > > > > > separate
>> > > > > > > > > > > > > > > config
>> > > > > > > > > > > > > > > > is probably the most natural way to support
>> it
>> > in
>> > > > the
>> > > > > > > > Polaris
>> > > > > > > > > > > > > codebase.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > The alternative, I guess, is to add R2
>> > accountId
>> > > > and
>> > > > > > > > > > jurisdiction
>> > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > optional properties to
>> > AwsStorageConfigurationInfo
>> > > > > (or
>> > > > > > > > > overload
>> > > > > > > > > > > > > > existing
>> > > > > > > > > > > > > > > > properties), but that looks like mixing
>> > unrelated
>> > > > > > > concepts
>> > > > > > > > > > > > together.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Adding a generic property bag to
>> > > > > > > > AwsStorageConfigurationInfo
>> > > > > > > > > is
>> > > > > > > > > > > not
>> > > > > > > > > > > > > > > > convenient because
>> > > > > > PolarisStorageIntegrationProviderImpl
>> > > > > > > > will
>> > > > > > > > > > > > > probably
>> > > > > > > > > > > > > > > have
>> > > > > > > > > > > > > > > > to know how to interpret it in order to
>> > redirect
>> > > > > > > credential
>> > > > > > > > > > > vending
>> > > > > > > > > > > > > > > > requests to R2-specific code.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > I wonder whether R2StorageConfigurationInfo
>> and
>> > > > > > > > > > > > > > > AwsStorageConfigurationInfo
>> > > > > > > > > > > > > > > > (and corresponding "storage integration"
>> > classes)
>> > > > > could
>> > > > > > > > > share a
>> > > > > > > > > > > > > common
>> > > > > > > > > > > > > > > base
>> > > > > > > > > > > > > > > > class for common properties (e.g. endpoint).
>> > Even
>> > > > > > though
>> > > > > > > > > > > "endpoint"
>> > > > > > > > > > > > > > might
>> > > > > > > > > > > > > > > > not be relevant to R2 in the cloud, the fact
>> > that
>> > > > > > > S3FileIO
>> > > > > > > > > uses
>> > > > > > > > > > > it,
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > R2
>> > > > > > > > > > > > > > > > is accessed via S3FileIO means we should
>> > probably
>> > > > > allow
>> > > > > > > > users
>> > > > > > > > > > to
>> > > > > > > > > > > > > > > configure
>> > > > > > > > > > > > > > > > the full set of FileIO properties just in
>> case.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > I think it is fine for different
>> > > > > > > PolarisStorageIntegration
>> > > > > > > > > > > > > > > implementations
>> > > > > > > > > > > > > > > > to produce the same kind of
>> > StorageAccessConfig (S3
>> > > > > in
>> > > > > > > this
>> > > > > > > > > > case)
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > > > thus
>> > > > > > > > > > > > > > > > cause S3FileIO to be used by clients
>> (including
>> > > > > > Polaris'
>> > > > > > > > own
>> > > > > > > > > > > > storage
>> > > > > > > > > > > > > > > access
>> > > > > > > > > > > > > > > > paths).
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Changes in StorageTypeFileIO in [1] look a
>> bit
>> > > > > > concerning
>> > > > > > > > to
>> > > > > > > > > > me.
>> > > > > > > > > > > I
>> > > > > > > > > > > > > > wonder
>> > > > > > > > > > > > > > > > if this code could be refactored to avoid
>> > > > > dependencies
>> > > > > > on
>> > > > > > > > > > storage
>> > > > > > > > > > > > > > config
>> > > > > > > > > > > > > > > > types completely.... but TBH, I did not look
>> > too
>> > > > > deeply
>> > > > > > > > into
>> > > > > > > > > > this
>> > > > > > > > > > > > > > today.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Anticipating future PRs for this in the
>> Polaris
>> > > > repo,
>> > > > > > I'd
>> > > > > > > > > like
>> > > > > > > > > > to
>> > > > > > > > > > > > ask
>> > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > separate Python code changes from java
>> changes
>> > > > since
>> > > > > > > these
>> > > > > > > > > > areas
>> > > > > > > > > > > > > > usually
>> > > > > > > > > > > > > > > > attract different reviewers :)
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > From my POV, management API and matching
>> java
>> > > > changes
>> > > > > > can
>> > > > > > > > be
>> > > > > > > > > in
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > same
>> > > > > > > > > > > > > > > > PR. I'd say docs should come later (to keep
>> PRs
>> > > > > small).
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > All in all, thank you again and I hope this
>> > > > > > contribution
>> > > > > > > > > lands
>> > > > > > > > > > in
>> > > > > > > > > > > > > > Polaris
>> > > > > > > > > > > > > > > > :)
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > [1]
>> > https://github.com/deepdishgary/polaris/pull/1
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Cheers,
>> > > > > > > > > > > > > > > > Dmitri.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > On Thu, Sep 10, 2026 at 1:52 PM
>> Jean-Baptiste
>> > > > Onofré
>> > > > > <
>> > > > > > > > > > > > > jb@nanthrax.net>
>> > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Hi Austen,
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Nice discussion! Thanks for that, and
>> welcome
>> > > > > aboard
>> > > > > > :)
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > I think it makes sense to have R2 built
>> > under the
>> > > > > > > > existing
>> > > > > > > > > S3
>> > > > > > > > > > > > > storage
>> > > > > > > > > > > > > > > > > configuration. It matches how Polaris is
>> > already
>> > > > > > > > > structured:
>> > > > > > > > > > > > > > > > > StorageType is keyed on the URI scheme
>> > (s3://,
>> > > > > > s3a://),
>> > > > > > > > and
>> > > > > > > > > > > > > > > > > AwsStorageConfigurationInfo already
>> carries
>> > > > > endpoint,
>> > > > > > > > > > > > stsEndpoint,
>> > > > > > > > > > > > > > > > > pathStyleAccess and, importantly, an
>> > > > stsUnavailable
>> > > > > > > flag.
>> > > > > > > > > > > > > > > > > So, what you need is largely there
>> already:
>> > > > correct
>> > > > > > me
>> > > > > > > if
>> > > > > > > > > I'm
>> > > > > > > > > > > > > wrong,
>> > > > > > > > > > > > > > > > > R2 is essentially "S3 compatible store
>> where
>> > STS
>> > > > is
>> > > > > > > > > > > unavailable,
>> > > > > > > > > > > > > plus
>> > > > > > > > > > > > > > > > > a way to actually vend credentials in that
>> > case".
>> > > > > > Going
>> > > > > > > > > this
>> > > > > > > > > > > > route
>> > > > > > > > > > > > > > > > > also keeps this work "isolated" from the
>> > > > > > > > > > > FileIO-per-storage-type
>> > > > > > > > > > > > > > > > > discussion.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > The interesting design question is a
>> > pluggable
>> > > > > > > credential
>> > > > > > > > > > > vending
>> > > > > > > > > > > > > > path
>> > > > > > > > > > > > > > > > > for S3 compatible stores when STS is not
>> > > > available,
>> > > > > > > with
>> > > > > > > > > > > > > Cloudflare's
>> > > > > > > > > > > > > > > > > local signing as the first concrete
>> > > > implementation,
>> > > > > > > > rather
>> > > > > > > > > > than
>> > > > > > > > > > > > R2
>> > > > > > > > > > > > > > > > > specific code. It would be good to sketch
>> how
>> > > > MinIO
>> > > > > > > would
>> > > > > > > > > > slot
>> > > > > > > > > > > > into
>> > > > > > > > > > > > > > > > > the same config fields before the vending
>> > path
>> > > > > > hardens
>> > > > > > > > > around
>> > > > > > > > > > > R2.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > I believe we have to check if Cloudflare
>> > account
>> > > > ID
>> > > > > > and
>> > > > > > > > > > > > > jurisdiction
>> > > > > > > > > > > > > > > > > really need dedicated config fields, or
>> can
>> > we
>> > > > use
>> > > > > > the
>> > > > > > > > > > endpoint
>> > > > > > > > > > > > > value
>> > > > > > > > > > > > > > > > > (with documentation)? Keeping the
>> > configuration
>> > > > > > surface
>> > > > > > > > > > minimal
>> > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > help the design generalize.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > I would love to discuss two things:
>> > > > > > > > > > > > > > > > > 1. Polaris shipping a Cloudflare API token
>> > and
>> > > > > > locally
>> > > > > > > > > > minting
>> > > > > > > > > > > > > scoped
>> > > > > > > > > > > > > > > > > credentials makes it effectively an STS
>> for
>> > R2,
>> > > > > which
>> > > > > > > > > > increases
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > blast radius of a server compromise. Your
>> > "reject
>> > > > > > > rather
>> > > > > > > > > than
>> > > > > > > > > > > > > > boarden"
>> > > > > > > > > > > > > > > > > handling of unrepresentable scopes is the
>> > right
>> > > > > call.
>> > > > > > > > > > > > > > > > > 2. The PyIceberg/DuckDB/Java results are
>> > > > promising.
>> > > > > > > Just
>> > > > > > > > > > > curious
>> > > > > > > > > > > > > > about
>> > > > > > > > > > > > > > > > > Spark or Trino.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > We love small, well-scoped PRs. I would
>> > suggest
>> > > > to
>> > > > > > > start
>> > > > > > > > > with
>> > > > > > > > > > > the
>> > > > > > > > > > > > > S3
>> > > > > > > > > > > > > > > > > config and the management API as first
>> > shoot, and
>> > > > > we
>> > > > > > > can
>> > > > > > > > > > follow
>> > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > the credential vending implementation.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Regards
>> > > > > > > > > > > > > > > > > JB
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > On Wed, Sep 9, 2026 at 8:18 PM Austen
>> Tomek
>> > > > > > > > > > > > > > > > > <Au...@chicagotrading.com> wrote:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Hi all,
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > I’m Austen Tomek from Chicago Trading
>> > Company.
>> > > > We
>> > > > > > use
>> > > > > > > > > > Polaris
>> > > > > > > > > > > > for
>> > > > > > > > > > > > > > our
>> > > > > > > > > > > > > > > > > Iceberg catalogs and we're evaluating
>> > Cloudflare
>> > > > R2
>> > > > > > as
>> > > > > > > an
>> > > > > > > > > > > object
>> > > > > > > > > > > > > > store
>> > > > > > > > > > > > > > > > (the
>> > > > > > > > > > > > > > > > > no egress fees make it a very compelling
>> > > > product).
>> > > > > > I’d
>> > > > > > > > like
>> > > > > > > > > > to
>> > > > > > > > > > > > > > discuss
>> > > > > > > > > > > > > > > > > contributing R2 support and credential
>> > vending to
>> > > > > > > Polaris
>> > > > > > > > > > with
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > goal
>> > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > getting feedback before opening a
>> > > > ready-for-review
>> > > > > > > > upstream
>> > > > > > > > > > PR.
>> > > > > > > > > > > > > This
>> > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > my
>> > > > > > > > > > > > > > > > > first contribution, so trying to do this
>> > right.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > R2 exposes S3-compatible APIs but it
>> does
>> > not
>> > > > > > provide
>> > > > > > > > AWS
>> > > > > > > > > > > STS.
>> > > > > > > > > > > > > > > > > Cloudflare instead supports these
>> > short-lived,
>> > > > > scoped
>> > > > > > > > > > > credentials
>> > > > > > > > > > > > > > > > generated
>> > > > > > > > > > > > > > > > > by locally signed JWT with the server
>> > holding the
>> > > > > > > parent
>> > > > > > > > > API
>> > > > > > > > > > > > token
>> > > > > > > > > > > > > > [1].
>> > > > > > > > > > > > > > > > > This allows us to retain Polaris’s
>> > authorization
>> > > > > and
>> > > > > > > > > > > > > > credential-vending
>> > > > > > > > > > > > > > > > > model while storing our table data in R2.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > I have a prototype with the following
>> > design:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >   *
>> > > > > > > > > > > > > > > > > > An opt-in R2 storage type, with catalog
>> > > > > > configuration
>> > > > > > > > > > > > identifying
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > Cloudflare account and optional
>> jurisdiction.
>> > > > > Parent
>> > > > > > > > > > > credentials
>> > > > > > > > > > > > > > remain
>> > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > server configuration and can be selected
>> > through
>> > > > > > > > > storageName.
>> > > > > > > > > > > > > > > > > >   *
>> > > > > > > > > > > > > > > > > > Local credential generation using
>> > Cloudflare’s
>> > > > > > > > documented
>> > > > > > > > > > > > format.
>> > > > > > > > > > > > > > > > > Credentials expire, are restricted to one
>> > bucket,
>> > > > > and
>> > > > > > > > carry
>> > > > > > > > > > > > prefix
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > access scopes derived from Polaris’s
>> location
>> > > > > grants.
>> > > > > > > The
>> > > > > > > > > > > > > integration
>> > > > > > > > > > > > > > > > uses
>> > > > > > > > > > > > > > > > > Polaris’s existing credential cache.
>> > > > > > > > > > > > > > > > > >   *
>> > > > > > > > > > > > > > > > > > Credentials returned through the
>> existing
>> > > > Iceberg
>> > > > > > > REST
>> > > > > > > > > > > vending
>> > > > > > > > > > > > > flow
>> > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > s3.* properties, including the endpoint,
>> > session
>> > > > > > token,
>> > > > > > > > and
>> > > > > > > > > > > > expiry.
>> > > > > > > > > > > > > > > > > Compatible clients use their existing S3
>> > FileIO.
>> > > > > > > > > > > > > > > > > >   *
>> > > > > > > > > > > > > > > > > > Conservative handling of scopes the
>> > > > > implementation
>> > > > > > > > cannot
>> > > > > > > > > > > > > > represent:
>> > > > > > > > > > > > > > > > > cross-bucket and mixed read/write grants
>> are
>> > > > > rejected
>> > > > > > > > > rather
>> > > > > > > > > > > than
>> > > > > > > > > > > > > > > > > broadened. R2 is excluded from the default
>> > > > > > > > > > > > supported-storage-types
>> > > > > > > > > > > > > > > list.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > The main design questions I feel this
>> runs
>> > > > into:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >   *
>> > > > > > > > > > > > > > > > > > Should this be a distinct storage type
>> or a
>> > > > > > > > > > > credential-vending
>> > > > > > > > > > > > > > > > mechanism
>> > > > > > > > > > > > > > > > > selected through the existing S3
>> > configuration?
>> > > > > > > > > > > > > > > > > >      *
>> > > > > > > > > > > > > > > > > > I chose a separate type because the
>> > identity,
>> > > > > > > > credential
>> > > > > > > > > > > > > > generation,
>> > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > scoping model differ from AWS STS, as this
>> > seemed
>> > > > > > more
>> > > > > > > > > > > > appropriate
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > There is a draft preview PR against my
>> fork
>> > > > [2],
>> > > > > > > > > including
>> > > > > > > > > > > > > > > > > implementation, management API changes,
>> CLI
>> > > > > support,
>> > > > > > > > tests,
>> > > > > > > > > > and
>> > > > > > > > > > > > > > > > > documentation. It also discloses the AI
>> > > > assistance
>> > > > > > used
>> > > > > > > > > > during
>> > > > > > > > > > > > > > > > > implementation. I leaned heavily on Fable
>> > 5.1 for
>> > > > > > this
>> > > > > > > > > > > > > implementation
>> > > > > > > > > > > > > > > so
>> > > > > > > > > > > > > > > > I
>> > > > > > > > > > > > > > > > > want to be upfront about it.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > We are running the implementation in a
>> > > > > > non-production
>> > > > > > > > > > > > environment
>> > > > > > > > > > > > > > > right
>> > > > > > > > > > > > > > > > > now as a POC. Validation includes live R2
>> > > > > > reads/writes,
>> > > > > > > > > > > multipart
>> > > > > > > > > > > > > > > > > operations, negative scope-boundary tests,
>> > our
>> > > > > > internal
>> > > > > > > > > > Python
>> > > > > > > > > > > > > client
>> > > > > > > > > > > > > > > > > regression matrix, and credential
>> > > > rotation/recovery
>> > > > > > > > > > exercises.
>> > > > > > > > > > > > The
>> > > > > > > > > > > > > > > > upstream
>> > > > > > > > > > > > > > > > > CI workflow has passed on the fork.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > The change also encounters the existing
>> > > > > assumption
>> > > > > > > > that a
>> > > > > > > > > > > > FileIO
>> > > > > > > > > > > > > > > > > implementation identifies exactly one
>> storage
>> > > > type:
>> > > > > > > both
>> > > > > > > > S3
>> > > > > > > > > > and
>> > > > > > > > > > > > R2
>> > > > > > > > > > > > > > use
>> > > > > > > > > > > > > > > > > S3FileIO. This overlaps with #4486 [3].
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > I’d particularly appreciate feedback on:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >   1.
>> > > > > > > > > > > > > > > > > > Should R2 have its own storage
>> > > > > configuration/type,
>> > > > > > or
>> > > > > > > > > > should
>> > > > > > > > > > > > > > non-STS
>> > > > > > > > > > > > > > > > > vending be modeled within the existing S3
>> > > > > > integration?
>> > > > > > > > > > > > > > > > > >   2.
>> > > > > > > > > > > > > > > > > > Does the local-signing approach fit
>> > Polaris’s
>> > > > > > > > > > > > credential-vending
>> > > > > > > > > > > > > > > model?
>> > > > > > > > > > > > > > > > > Any additional constraints or validations
>> > > > > necessary?
>> > > > > > > > > > > > > > > > > >   3.
>> > > > > > > > > > > > > > > > > > Would you prefer the shared FileIO
>> > validation
>> > > > > > change
>> > > > > > > to
>> > > > > > > > > > land
>> > > > > > > > > > > > > > > > separately,
>> > > > > > > > > > > > > > > > > and how should we sequence the management
>> > API and
>> > > > > R2
>> > > > > > > > > > > integration
>> > > > > > > > > > > > > > > changes?
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >   Happy to adapt the design and split
>> the
>> > > > > > > contribution
>> > > > > > > > > into
>> > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > focused
>> > > > > > > > > > > > > > > > > PRs. Wanted to start the discussion
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >   Thanks,
>> > > > > > > > > > > > > > > > > >   Austen
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >   [1]
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > >
>> https://developers.cloudflare.com/r2/api/s3/temporary-credentials/
>> > > > > > > > > > > > > > > > > >   [2]
>> > > > > > https://github.com/deepdishgary/polaris/pull/1
>> > > > > > > > > > > > > > > > > >   [3]
>> > > > > > https://github.com/apache/polaris/issues/4486
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Get Outlook for Mac<
>> > > > > > https://aka.ms/GetOutlookForMac>
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > This electronic mail message and any
>> > attached
>> > > > > files
>> > > > > > > > > contain
>> > > > > > > > > > > > > > > information
>> > > > > > > > > > > > > > > > > intended for the exclusive use of the
>> > individual
>> > > > or
>> > > > > > > > entity
>> > > > > > > > > to
>> > > > > > > > > > > > whom
>> > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > addressed and may contain information
>> that is
>> > > > > > > > proprietary,
>> > > > > > > > > > > > > > confidential
>> > > > > > > > > > > > > > > > > and/or exempt from disclosure under
>> > applicable
>> > > > law.
>> > > > > > If
>> > > > > > > > you
>> > > > > > > > > > are
>> > > > > > > > > > > > not
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > intended recipient, you are hereby
>> notified
>> > that
>> > > > > any
>> > > > > > > > > viewing,
>> > > > > > > > > > > > > > copying,
>> > > > > > > > > > > > > > > > > disclosure or distribution of this
>> > information
>> > > > may
>> > > > > be
>> > > > > > > > > subject
>> > > > > > > > > > > to
>> > > > > > > > > > > > > > legal
>> > > > > > > > > > > > > > > > > restriction or sanction. Please notify the
>> > > > sender,
>> > > > > by
>> > > > > > > > > > > electronic
>> > > > > > > > > > > > > mail
>> > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > telephone, of any unintended recipients
>> and
>> > > > delete
>> > > > > > the
>> > > > > > > > > > original
>> > > > > > > > > > > > > > message
>> > > > > > > > > > > > > > > > > without making any copies.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > This electronic mail message and any attached
>> files
>> > > > > contain
>> > > > > > > > > > > information
>> > > > > > > > > > > > > > intended for the exclusive use of the
>> individual or
>> > > > > entity
>> > > > > > to
>> > > > > > > > > whom
>> > > > > > > > > > it
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > addressed and may contain information that is
>> > > > > proprietary,
>> > > > > > > > > > > confidential
>> > > > > > > > > > > > > > and/or exempt from disclosure under applicable
>> > law. If
>> > > > > you
>> > > > > > > are
>> > > > > > > > > not
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > intended recipient, you are hereby notified that
>> > any
>> > > > > > viewing,
>> > > > > > > > > > > copying,
>> > > > > > > > > > > > > > disclosure or distribution of this information
>> may
>> > be
>> > > > > > subject
>> > > > > > > > to
>> > > > > > > > > > > legal
>> > > > > > > > > > > > > > restriction or sanction. Please notify the
>> sender,
>> > by
>> > > > > > > > electronic
>> > > > > > > > > > mail
>> > > > > > > > > > > > or
>> > > > > > > > > > > > > > telephone, of any unintended recipients and
>> delete
>> > the
>> > > > > > > original
>> > > > > > > > > > > message
>> > > > > > > > > > > > > > without making any copies.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > This electronic mail message and any attached files
>> > contain
>> > > > > > > > information
>> > > > > > > > > > > intended for the exclusive use of the individual or
>> > entity to
>> > > > > > whom
>> > > > > > > it
>> > > > > > > > > is
>> > > > > > > > > > > addressed and may contain information that is
>> > proprietary,
>> > > > > > > > confidential
>> > > > > > > > > > > and/or exempt from disclosure under applicable law. If
>> > you
>> > > > are
>> > > > > > not
>> > > > > > > > the
>> > > > > > > > > > > intended recipient, you are hereby notified that any
>> > viewing,
>> > > > > > > > copying,
>> > > > > > > > > > > disclosure or distribution of this information may be
>> > subject
>> > > > > to
>> > > > > > > > legal
>> > > > > > > > > > > restriction or sanction. Please notify the sender, by
>> > > > > electronic
>> > > > > > > mail
>> > > > > > > > > or
>> > > > > > > > > > > telephone, of any unintended recipients and delete the
>> > > > original
>> > > > > > > > message
>> > > > > > > > > > > without making any copies.
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > This electronic mail message and any attached files
>> contain
>> > > > > > > information
>> > > > > > > > > > intended for the exclusive use of the individual or
>> entity
>> > to
>> > > > > whom
>> > > > > > it
>> > > > > > > > is
>> > > > > > > > > > addressed and may contain information that is
>> proprietary,
>> > > > > > > confidential
>> > > > > > > > > > and/or exempt from disclosure under applicable law. If
>> you
>> > are
>> > > > > not
>> > > > > > > the
>> > > > > > > > > > intended recipient, you are hereby notified that any
>> > viewing,
>> > > > > > > copying,
>> > > > > > > > > > disclosure or distribution of this information may be
>> > subject
>> > > > to
>> > > > > > > legal
>> > > > > > > > > > restriction or sanction. Please notify the sender, by
>> > > > electronic
>> > > > > > mail
>> > > > > > > > or
>> > > > > > > > > > telephone, of any unintended recipients and delete the
>> > original
>> > > > > > > message
>> > > > > > > > > > without making any copies.
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > This electronic mail message and any attached files contain
>> > > > > information
>> > > > > > > > intended for the exclusive use of the individual or entity
>> to
>> > whom
>> > > > it
>> > > > > > is
>> > > > > > > > addressed and may contain information that is proprietary,
>> > > > > confidential
>> > > > > > > > and/or exempt from disclosure under applicable law. If you
>> are
>> > not
>> > > > > the
>> > > > > > > > intended recipient, you are hereby notified that any
>> viewing,
>> > > > > copying,
>> > > > > > > > disclosure or distribution of this information may be
>> subject
>> > to
>> > > > > legal
>> > > > > > > > restriction or sanction. Please notify the sender, by
>> > electronic
>> > > > mail
>> > > > > > or
>> > > > > > > > telephone, of any unintended recipients and delete the
>> original
>> > > > > message
>> > > > > > > > without making any copies.
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> >
>>
>

Re: [DISCUSS] Cloudflare R2 support with scoped credential vending

[Dmitri Bourlatchkov <di...@apache.org>]

HI Yufei,

Yes, the proposed [5513] Management REST API change lists possible values
in Open API doc fields on the affected types [1].

ATM, it mentions "STS" as the only supported option. We can certainly add
"R2" to that list when it is implemented.

[1]
https://github.com/apache/polaris/pull/5513/changes#diff-52444bc79608edfae86ed0b46d171f7ef63c20090860d877e4e135168311a986R1211-R1213

[5513] https://github.com/apache/polaris/pull/5513

Cheers,
Dmitri.

On Thu, Oct 1, 2026 at 1:52 PM Yufei Gu <fl...@gmail.com> wrote:

> Thanks for the summary, Dmitri!
>
> +1 on the initial list ("STS" and "R2"). These will be written to the REST
> spec along with the new key introduced (credentialVendingMechanism).
>
> Yufei
>
>
> On Thu, Oct 1, 2026 at 10:14 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > Recap from today's community sync call:
> >
> > * Actual set of values acceptable for "credentialVendingMechanism" is a
> > finite list. Initially "STS" and "R2" will be supported by Polaris (exact
> > names TBD during R2 implementation).
> >
> > * Servers validate values in "credentialVendingMechanism" in runtime
> after
> > parsing the request.
> >
> > * The Open API doc defines "credentialVendingMechanism" as a string to
> > allow for flexibility in auto-generated code. Clients / parsers do not
> have
> > to re-build when a server adds a new option for
> > "credentialVendingMechanism".
> >
> > Please correct me if I missed anything.
> >
> > Does this sound acceptable?
> >
> > Thanks,
> > Dmitri.
> >
> > On Wed, Sep 30, 2026 at 1:30 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Yufei,
> > >
> > > Thanks for providing the details! Replying inline.
> > >
> > > > When a client creates a catalog with credentialVendingMechanism:
> "m1",
> > > > what is the expected behavior?
> > >
> > > This depends on the meaning of "m1" and the specific server build.
> > >
> > > In Apache Polaris "m1" is not meaningful and is not supported.
> > > The server will reject this value on the corresponding catalog
> > > update request. This is handled in PolarisAdminService in the current
> PR
> > > state.
> > >
> > > > 1.  The server supports it: This is the ideal scenario (happy path).
> > >
> > > Indeed.
> > >
> > > > 2.  The server does not support it: This creates inconsistency if a
> > > client
> > > > that works with Server A fails on Server B.
> > >
> > > In my view this is not an inconsistency as different servers do not
> have
> > > to provide the same capabilities.
> > >
> > > Creating a catalog with a specific credentialVendingMechanism is an
> > > administrative task. The admin user must be aware of what the server
> > > supports. This is a deployment / configuration concern. Not a runtime
> > > compatibility concern.
> > >
> > > > 3.  The server supports "m1" differently: This is the worst-case
> > > scenario,
> > > > where the client's expectations conflict with the server's actual
> > > behavior.
> > >
> > > This is a good point!
> > >
> > > My take is that if "m1" is defined in Apache Polaris, then all server
> > > builds must implement it the same way.
> > >
> > > The definition will be expressed by means of
> S3CredentialVendingMechanism
> > > classes included in standard Apache Polaris releases (and hopefully
> docs
> > > too).
> > >
> > > Currently only "STS" and "DEFAULT" (falling back on STS) are defined
> now.
> > >
> > > If "m1" is not defined in Apache Polaris, custom builds are free to
> > assign
> > > any meaning to it and do not have to be mutually compatible. This is
> the
> > > main point for extensibility.
> > >
> > > Note that custom builds are not "Apache Polaris" anymore and cannot
> > > claim to be the "same" as Apache Polaris. This is a matter of ASF
> > > trademarks.
> > > Normally, custom builds say something like "Powered by Apache Polaris".
> > >
> > > Again, the admin user setting "m1" as a vending mechanism in a
> particular
> > > server must be aware of what that server provides and understand
> > > the meaning of "m1" in that context. The onus to describe the meaning
> of
> > > "m1" to users in this case rests with the custom server developers.
> > >
> > > We can talk about preventing name overlaps, but I do not think
> > establishing
> > > any kind of name registry is worth the trouble. In this day and age, I
> > > believe
> > > all downstream project developers are aware of name overlap issues.
> > > and will follow normal practices to avoid name collisions.
> > >
> > > I hope this clears your concerns.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Wed, Sep 30, 2026 at 1:02 PM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > >> Hi Dmitri,
> > >>
> > >> When a client creates a catalog with credentialVendingMechanism: "m1",
> > >> what
> > >> is the expected behavior?
> > >>
> > >> Here is a suitaion if we allow a freeform string, the client might
> > >> encounter three different scenarios depending on the server:
> > >> 1.  The server supports it: This is the ideal scenario (happy path).
> > >> 2.  The server does not support it: This creates inconsistency if a
> > client
> > >> that works with Server A fails on Server B.
> > >> 3.  The server supports "m1" differently: This is the worst-case
> > scenario,
> > >> where the client's expectations conflict with the server's actual
> > >> behavior.
> > >>
> > >> I don't think this is the situation we want users to run into. The fix
> > is
> > >> quite simple, using a specified values instead of a freefrom string.
> > >>
> > >> Yufei
> > >>
> > >>
> > >> On Tue, Sep 29, 2026 at 9:48 PM Jean-Baptiste Onofré <jb@nanthrax.net
> >
> > >> wrote:
> > >>
> > >> > Hi all,
> > >> >
> > >> > Since concerns were raised, it's important to take a moment to align
> > >> > rather than rushing. That's the way we drive consensus :)
> > >> >
> > >> > On the technical question of string vs enum for
> > >> > credentialVendingMechanism, here is my take:
> > >> >
> > >> > 1. This field is part of the Management API for catalog
> > >> > administrators. Iceberg REST clients (Spark, Trino, ...) never see
> or
> > >> > interact with credentialVendingMechanism. They only consume standard
> > >> > vended credentials (s3.* session tokens/keys) during table
> > >> > load/commit. There is no risk of breaking query engine clients
> across
> > >> > deployments.
> > >> > 2. One of our primary goals when discussing S3-compatible stores
> (R2,
> > >> > MinIO, ...) was to keep credential vending pluggable via SPI rather
> > >> > than hardcoding vendor-specific paths in core. If we use a closed
> enum
> > >> > in the OpenAPI spec, it means:
> > >> >   * any new mechanism (including R2 in Phase 2 or MinIO) requires a
> > spec
> > >> > change.
> > >> >   * any downstream deployment or plugin providing a custom mechanism
> > >> > would break OpenAPI-generated client SDKs due to strict enum
> > >> > validation.
> > >> >   * runtime validation on the server (returning HTTP 400 with a
> clear
> > >> > message) combined with the realm allowlist provides the right
> > >> > server-side guardrail without restricting extensibility.
> > >> > 3. Yufei's concerns about a clear contract and avoiding confusion is
> > >> > valid. We can address contract clarity without closing the enum:
> > >> >   * we keep type: string, but document the well-known/standard
> values
> > >> > (STS, DEFAULT) explicitly in the description and examples in the
> > >> > OpenAPI spec (as PR #5513 does).
> > >> >   * we ensure the server validates cleanly with case-insensitivity
> (or
> > >> > canonical normalization) so administrators get immediate, actionable
> > >> > feedback on typos.
> > >> >
> > >> > PR #5513 does a solid job laying this foundation so Austen can move
> > >> > forward with Phase 2. If we keep it as a string with well-documented
> > >> > standard values and server-siide validation, I think it's the right
> > >> > balance between spec clarify and pluggability.
> > >> >
> > >> > Thoughts?
> > >> >
> > >> > Regards
> > >> > JB
> > >> >
> > >> > On Wed, Sep 30, 2026 at 12:50 AM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > >> > wrote:
> > >> > >
> > >> > > Hi Yufei,
> > >> > >
> > >> > > > > Conversely, an enum will require defining all possible vending
> > >> > > mechanisms in
> > >> > > > > the Polaris API spec.
> > >> > >
> > >> > > > Yes. Otherwise, a client may failed in certain deployments.
> > >> > >
> > >> > > Could you describe this failure mode in terms of API
> > request/response
> > >> > flow?
> > >> > >
> > >> > > Thanks,
> > >> > > Dmitri.
> > >> > >
> > >> > > On Tue, Sep 29, 2026 at 2:06 PM Yufei Gu <fl...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > > > > Conversely, an enum will require defining all possible vending
> > >> > > > mechanisms in
> > >> > > > the Polaris API spec.
> > >> > > >
> > >> > > > Yes. Otherwise, a client may failed in certain deployments.
> > >> > > >
> > >> > > > > This will prevent downstream projects from using custom
> vending
> > >> > > > mechanisms.
> > >> > > >
> > >> > > > No. if downstream needs custom vending mechanisms, they can
> always
> > >> > build
> > >> > > > their own Polaris.
> > >> > > >
> > >> > > > Yufei
> > >> > > >
> > >> > > >
> > >> > > > On Tue, Sep 29, 2026 at 11:01 AM Dmitri Bourlatchkov <
> > >> dimas@apache.org
> > >> > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Yufei,
> > >> > > > >
> > >> > > > > > Using an enum will clarify the REST spec contract.
> > >> > > > >
> > >> > > > > I'm not sure I understand your point. Current PR clearly
> > describes
> > >> > the
> > >> > > > > meaning of "credentialVendingMechanism" and lists options
> > >> available
> > >> > > > > out-of-the-box.
> > >> > > > >
> > >> > > > > What does an enum achieve, that (current) runtime validation
> > does
> > >> > not?
> > >> > > > >
> > >> > > > > Conversely, an enum will require defining all possible vending
> > >> > mechanisms
> > >> > > > > in the Polaris API spec. This will prevent downstream projects
> > >> from
> > >> > using
> > >> > > > > custom vending mechanisms.
> > >> > > > >
> > >> > > > > I believe Polaris as a project currently promotes downstream
> > >> > > > > customizability.
> > >> > > > >
> > >> > > > > Cheers,
> > >> > > > > Dmitri.
> > >> > > > >
> > >> > > > > On Tue, Sep 29, 2026 at 1:21 PM Yufei Gu <
> flyrain000@gmail.com>
> > >> > wrote:
> > >> > > > >
> > >> > > > > > I don't think we have consensus on the PR, esp. a client
> > should
> > >> NOT
> > >> > > > > depend
> > >> > > > > > on a specific deployment. Using an enum will clarify the
> REST
> > >> spec
> > >> > > > > > contract.
> > >> > > > > >
> > >> > > > > > Yufei
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Tue, Sep 29, 2026 at 7:46 AM Dmitri Bourlatchkov <
> > >> > dimas@apache.org>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi All,
> > >> > > > > > >
> > >> > > > > > > PR [5513] has been in review for quite some time. I
> believe
> > we
> > >> > have
> > >> > > > > > general
> > >> > > > > > > consensus on the Management API changes and java code
> > changes.
> > >> > > > > > >
> > >> > > > > > > The only point that may still have divergent opinions is
> > >> whether
> > >> > to
> > >> > > > use
> > >> > > > > > > free text or enum for "credentialVendingMechanism". I
> > >> personally
> > >> > > > > support
> > >> > > > > > > free text (rationale explained in previous comments). This
> > >> > > > conversation
> > >> > > > > > has
> > >> > > > > > > been dormant for a few days, so I assume lazy consensus on
> > the
> > >> > > > current
> > >> > > > > > > state of the PR (free text). In any case switching from
> free
> > >> > text to
> > >> > > > > enum
> > >> > > > > > > is a non-breaking API change before the next release in
> case
> > >> the
> > >> > > > > > consensus
> > >> > > > > > > shifts and we need to adjust. We have about one month for
> > >> > followup
> > >> > > > API
> > >> > > > > > > changes.
> > >> > > > > > >
> > >> > > > > > > The java code introduced in this PR is not rigid and can
> > >> change
> > >> > at
> > >> > > > any
> > >> > > > > > time
> > >> > > > > > > per our evolution guidelines [1].
> > >> > > > > > >
> > >> > > > > > > I propose merging PR [5513] on Sept 30 unless fresh
> blocking
> > >> > concerns
> > >> > > > > are
> > >> > > > > > > raised.
> > >> > > > > > >
> > >> > > > > > > [1] https://polaris.apache.org/releases/1.8.0/evolution/
> > >> > > > > > >
> > >> > > > > > > [5513] https://github.com/apache/polaris/pull/5513
> > >> > > > > > >
> > >> > > > > > > Cheers,
> > >> > > > > > > Dmitri.
> > >> > > > > > >
> > >> > > > > > > On Tue, Sep 29, 2026 at 9:30 AM Austen Tomek <
> > >> > > > > > > Austen.Tomek@chicagotrading.com> wrote:
> > >> > > > > > >
> > >> > > > > > > > Hey all,
> > >> > > > > > > >
> > >> > > > > > > > Any chance I can get further reviews this week on the
> PR?
> > >> I’d
> > >> > like
> > >> > > > to
> > >> > > > > > get
> > >> > > > > > > > moving onto Phase 2 here if there are no further issues
> to
> > >> > address.
> > >> > > > > > > >
> > >> > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac>
> > >> > > > > > > >
> > >> > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> > >> > > > > > > > Date: Thursday, September 24, 2026 at 17:33
> > >> > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > >> > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped
> > >> > credential
> > >> > > > > > > vending
> > >> > > > > > > >
> > >> > > > > > > > Hi Yufei,
> > >> > > > > > > >
> > >> > > > > > > > > The REST spec is a contract between Polaris servers
> and
> > >> > clients
> > >> > > > > [...]
> > >> > > > > > > >
> > >> > > > > > > > I tend to think that this statement is a bit too strong.
> > The
> > >> > > > > Management
> > >> > > > > > > API
> > >> > > > > > > > spec defines a protocol for clients and servers to
> > >> communicate.
> > >> > > > > > > >
> > >> > > > > > > > > [...] This would help avoid a situation where a
> Polaris
> > >> > client
> > >> > > > > works
> > >> > > > > > > > > only with certain deployments.
> > >> > > > > > > >
> > >> > > > > > > > Since Polaris has been developed as a customizable
> system,
> > >> a
> > >> > REST
> > >> > > > > API
> > >> > > > > > > > specification defined by Apache Polaris cannot control
> the
> > >> > > > behaviour
> > >> > > > > of
> > >> > > > > > > all
> > >> > > > > > > > possible downstream environments. The purpose of the
> spec
> > >> is to
> > >> > > > > ensure
> > >> > > > > > > > protocol compatibility.
> > >> > > > > > > >
> > >> > > > > > > > Note that the value of `credentialVendingMechanism` has
> no
> > >> > bearing
> > >> > > > on
> > >> > > > > > > > any logic clients need to execute. It is an opaque
> > parameter
> > >> > > > > controlled
> > >> > > > > > > > by the Polaris administrator.
> > >> > > > > > > >
> > >> > > > > > > > > The new credentialVendingMechanism field in the REST
> > spec
> > >> > looks
> > >> > > > > > > > > reasonable. My only suggestion is to use an enum
> instead
> > >> of a
> > >> > > > > string.
> > >> > > > > > > > [...]
> > >> > > > > > > >
> > >> > > > > > > > The above points are exactly the reason why the
> Management
> > >> API
> > >> > > > cannot
> > >> > > > > > > > enumerate all possible vending mechanisms.
> > >> > > > > > > >
> > >> > > > > > > > The same applies to storage types too. However, we do
> not
> > >> have
> > >> > any
> > >> > > > > > > > practical
> > >> > > > > > > > examples of 3rd party storage systems that go beyond
> > >> > S3/GCS/ADLS
> > >> > > > > (yet).
> > >> > > > > > > >
> > >> > > > > > > > Cheers,
> > >> > > > > > > > Dmitri.
> > >> > > > > > > >
> > >> > > > > > > > On Thu, Sep 24, 2026 at 4:38 PM Yufei Gu <
> > >> flyrain000@gmail.com
> > >> > >
> > >> > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Thanks Austen. It's fair that endpoints are not
> reliable
> > >> > given
> > >> > > > that
> > >> > > > > > > some
> > >> > > > > > > > > s3-compatible can use arbitrary hostnames.
> > >> > > > > > > > > I also think stsUnavailable is fine to keep as-is. I
> > don’t
> > >> > think
> > >> > > > we
> > >> > > > > > > need
> > >> > > > > > > > to
> > >> > > > > > > > > change its scope or repurpose it.
> > >> > > > > > > > >
> > >> > > > > > > > > The new credentialVendingMechanism field in the REST
> > spec
> > >> > looks
> > >> > > > > > > > reasonable.
> > >> > > > > > > > > My only suggestion is to use an enum instead of a
> > string.
> > >> The
> > >> > > > REST
> > >> > > > > > spec
> > >> > > > > > > > is
> > >> > > > > > > > > a contract between Polaris servers and clients, so I’d
> > >> > expect it
> > >> > > > to
> > >> > > > > > > > define
> > >> > > > > > > > > the valid values and their meanings rather than
> leaving
> > >> them
> > >> > up
> > >> > > > to
> > >> > > > > > each
> > >> > > > > > > > > deployment. This would help avoid a situation where a
> > >> Polaris
> > >> > > > > client
> > >> > > > > > > > works
> > >> > > > > > > > > only with certain deployments. Deployment-specific
> > >> mechanisms
> > >> > > > > > shouldn’t
> > >> > > > > > > > be
> > >> > > > > > > > > part of the standard REST contract; users who need a
> > >> custom
> > >> > > > > mechanism
> > >> > > > > > > can
> > >> > > > > > > > > always extend the server in their own deployment.
> > >> > > > > > > > >
> > >> > > > > > > > > > Either way the operator needs a per-realm switch
> like
> > >> > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control
> > >> which
> > >> > realm
> > >> > > > > can
> > >> > > > > > > > mint
> > >> > > > > > > > > R2 credentials from the operators Cloudflare account.
> > >> > > > > > > > >
> > >> > > > > > > > > Could you clarify why we need a separate per-realm
> > switch?
> > >> > Once
> > >> > > > the
> > >> > > > > > > > storage
> > >> > > > > > > > > configuration is set at the catalog level to select a
> > >> > specific
> > >> > > > > > > mechanism,
> > >> > > > > > > > > the catalog will have no choice but to support it. Do
> we
> > >> need
> > >> > > > > another
> > >> > > > > > > > layer
> > >> > > > > > > > > of gating?
> > >> > > > > > > > >
> > >> > > > > > > > > Yufei
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Wed, Sep 23, 2026 at 5:53 AM Austen Tomek <
> > >> > > > > > > > > Austen.Tomek@chicagotrading.com> wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hey Yufei,
> > >> > > > > > > > > >
> > >> > > > > > > > > > Apologies for missing your question.
> > >> > > > > > > > > >
> > >> > > > > > > > > > I don’t think Endpoint is sufficient.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Endpoint matching only works where the hostname
> > >> identifies
> > >> > the
> > >> > > > > > vendor
> > >> > > > > > > > > like
> > >> > > > > > > > > > R2’s public hostnames that are recognizable, MinIO
> and
> > >> > other
> > >> > > > > > > > self-hosted
> > >> > > > > > > > > > S3-compatible stores use arbitrary hostnames so
> there
> > >> > would be
> > >> > > > > > > nothing
> > >> > > > > > > > to
> > >> > > > > > > > > > match on. On top of that, if a match fails it would
> > just
> > >> > > > silently
> > >> > > > > > > fall
> > >> > > > > > > > > back
> > >> > > > > > > > > > to STS.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Right now the endpoint just says how you’ll connect
> > not
> > >> > how the
> > >> > > > > > > > > > credentials are vended. There's endpoint and also
> > >> > > > > endpointInternal
> > >> > > > > > > for
> > >> > > > > > > > > > server-side access, so it would be unclear which of
> > the
> > >> two
> > >> > > > > > decides.
> > >> > > > > > > > > >
> > >> > > > > > > > > > For your other question, there is no second
> mechanism
> > >> for
> > >> > R2
> > >> > > > > today,
> > >> > > > > > > but
> > >> > > > > > > > > > for other self-hosted stores the same endpoint could
> > be
> > >> > served
> > >> > > > by
> > >> > > > > > STS
> > >> > > > > > > > or
> > >> > > > > > > > > by
> > >> > > > > > > > > > a different token mechanism chosen per catalog.
> > >> > > > > > > > > >
> > >> > > > > > > > > > stsUnavailable keeps it’s current behavior in the PR
> > >> (that
> > >> > was
> > >> > > > > > > > Prithvi’s
> > >> > > > > > > > > > point earlier in the thread)
> > >> > > > > > > > > >
> > >> > > > > > > > > > Either way the operator needs a per-realm switch
> like
> > >> > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to
> control
> > >> which
> > >> > > > realm
> > >> > > > > > can
> > >> > > > > > > > > mint
> > >> > > > > > > > > > R2 credentials from the operators Cloudflare
> account.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Austen
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac
> >
> > >> > > > > > > > > >
> > >> > > > > > > > > > From: Yufei Gu <fl...@gmail.com>
> > >> > > > > > > > > > Date: Monday, September 21, 2026 at 17:53
> > >> > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > >> > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with
> > scoped
> > >> > > > > credential
> > >> > > > > > > > > vending
> > >> > > > > > > > > >
> > >> > > > > > > > > > Hi Austen,
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks for the update. After looking at PR #5513, I
> > >> think
> > >> > we
> > >> > > > > should
> > >> > > > > > > > first
> > >> > > > > > > > > > clarify whether we need a new
> > credentialVendingMechanism
> > >> > field
> > >> > > > in
> > >> > > > > > the
> > >> > > > > > > > > > spec/config.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Austen, you mentioned that the R2 account ID and
> > >> > jurisdiction
> > >> > > > > > already
> > >> > > > > > > > > come
> > >> > > > > > > > > > from the endpoint. Could we also use the existing
> > field
> > >> > like
> > >> > > > > > > > `endpoint`,
> > >> > > > > > > > > to
> > >> > > > > > > > > > recognize R2 and select its credential-vending
> > >> mechanism,
> > >> > while
> > >> > > > > > > keeping
> > >> > > > > > > > > the
> > >> > > > > > > > > > current stsUnavailable behavior unchanged?
> > >> > > > > > > > > >
> > >> > > > > > > > > > Are there cases where the endpoint isn’t enough, or
> > >> where
> > >> > users
> > >> > > > > > need
> > >> > > > > > > to
> > >> > > > > > > > > > choose different vending mechanisms for the same
> > >> > endpoint(e.g.,
> > >> > > > > R2)
> > >> > > > > > > per
> > >> > > > > > > > > > catalog?
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Wed, Sep 16, 2026 at 7:24 AM Austen Tomek <
> > >> > > > > > > > > > Austen.Tomek@chicagotrading.com> wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi Yufei, Dimitri,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks, both. PR 1 is ready for review:
> > >> > > > > > > > > > > https://github.com/apache/polaris/pull/5513
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I pushed a revision last night that takes in
> > Dimitri’s
> > >> > > > > comments.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >   *
> > >> > > > > > > > > > > I changed from issuer to
> credentialVendingMechanism.
> > >> It’s
> > >> > > > also
> > >> > > > > > now
> > >> > > > > > > a
> > >> > > > > > > > > > plain
> > >> > > > > > > > > > > string and the realm allowlist is
> > >> > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS, which
> > >> still
> > >> > > > > defaults
> > >> > > > > > to
> > >> > > > > > > > > [STS]
> > >> > > > > > > > > > >   *
> > >> > > > > > > > > > > Mechanisms are CDI beans found by their
> identifier,
> > as
> > >> > > > Dimitri
> > >> > > > > > > > > proposed.
> > >> > > > > > > > > > >   *
> > >> > > > > > > > > > > The CLOUDFLARE_R2 bean and the R2 vending code
> will
> > >> > follow in
> > >> > > > > the
> > >> > > > > > > > next
> > >> > > > > > > > > PR
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > CI workflow is waiting approval to run.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > Austen
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Get Outlook for Mac<
> https://aka.ms/GetOutlookForMac
> > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> > >> > > > > > > > > > > Date: Tuesday, September 15, 2026 at 20:00
> > >> > > > > > > > > > > To: dev@polaris.apache.org <
> dev@polaris.apache.org>
> > >> > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with
> > >> scoped
> > >> > > > > > credential
> > >> > > > > > > > > > vending
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Hi All,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > Having getCloudflareR2Endpoint on
> > >> > > > AwsStorageConfigurationInfo
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Yes, this does feel awkward.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > From my POV pursuing the option with the
> > >> > "credentialIssuers"
> > >> > > > > > > property
> > >> > > > > > > > > > looks
> > >> > > > > > > > > > > promising. This property is fairly generic and
> fits
> > >> > naturally
> > >> > > > > > into
> > >> > > > > > > > the
> > >> > > > > > > > > > > config, given that we already target multiple
> > >> backends.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Cheers,
> > >> > > > > > > > > > > Dmitri.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Tue, Sep 15, 2026 at 8:48 PM Yufei Gu <
> > >> > > > flyrain000@gmail.com
> > >> > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > Hi Austen, Dmitri,
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > Having getCloudflareR2Endpoint on
> > >> > > > > AwsStorageConfigurationInfo
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > That does feel awkward. However,
> > >> > > > AwsStorageConfigurationInfo
> > >> > > > > > has
> > >> > > > > > > > > > > supported
> > >> > > > > > > > > > > > S3-compatible backends beyond AWS for a while,
> so
> > >> > there is
> > >> > > > > > > > precedent
> > >> > > > > > > > > > > (e.g.,
> > >> > > > > > > > > > > > MinIO, etc.). The underlying mismatch is that we
> > >> call
> > >> > the
> > >> > > > > > storage
> > >> > > > > > > > > type
> > >> > > > > > > > > > > S3,
> > >> > > > > > > > > > > > but its configuration mixes shared S3 settings
> > with
> > >> > > > > > AWS-specific
> > >> > > > > > > > > ones:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > AwsStorageConfigurationInfo → storage type: S3
> > >> > > > > > > > > > > >   ├── Common S3 settings
> > >> > > > > > > > > > > >   │     endpoint, region, pathStyleAccess,
> > >> > allowedLocations
> > >> > > > > > > > > > > >   └── AWS-specific settings
> > >> > > > > > > > > > > >         roleArn, externalId, STS, KMS
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > I’m open to separating the shared S3 settings
> from
> > >> > > > > > > > provider-specific
> > >> > > > > > > > > > > > configuration if that helps, but I don’t think
> we
> > >> need
> > >> > to
> > >> > > > > > settle
> > >> > > > > > > > on a
> > >> > > > > > > > > > > > broader refactor now.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Let me know when the PR is ready for review. I’d
> > be
> > >> > happy
> > >> > > > to
> > >> > > > > > > take a
> > >> > > > > > > > > > > closer
> > >> > > > > > > > > > > > look.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Yufei
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Yufei
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:18 PM Dmitri
> > Bourlatchkov
> > >> <
> > >> > > > > > > > > dimas@apache.org>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > Hi Austen,
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > +1 to adding "credentialIssues" or similar
> > >> (renamed)
> > >> > > > > property
> > >> > > > > > > to
> > >> > > > > > > > S3
> > >> > > > > > > > > > > > config.
> > >> > > > > > > > > > > > > In fact, I was looking exactly for something
> > like
> > >> > that
> > >> > > > last
> > >> > > > > > > week
> > >> > > > > > > > > but
> > >> > > > > > > > > > > the
> > >> > > > > > > > > > > > > idea did not quite condense in my mind :)
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > I posted some preliminary comments on the
> draft
> > >> PR.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Just for community awareness, I'd like to
> > propose
> > >> > here
> > >> > > > that
> > >> > > > > > we
> > >> > > > > > > > use
> > >> > > > > > > > > > CDI
> > >> > > > > > > > > > > > for
> > >> > > > > > > > > > > > > finding actual "issuer" / "mechanism"
> > >> > implementations.
> > >> > > > The
> > >> > > > > > > > codebase
> > >> > > > > > > > > > has
> > >> > > > > > > > > > > > > many examples of this. The closest match in
> this
> > >> > case is
> > >> > > > > > > > > > > > > probably PolarisStorageIntegrationProvider.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > This way the API does not have to enumerate
> all
> > >> > possible
> > >> > > > > > > values.
> > >> > > > > > > > > > > > Downstream
> > >> > > > > > > > > > > > > builds are free to add different
> implementations
> > >> > without
> > >> > > > > > having
> > >> > > > > > > > to
> > >> > > > > > > > > > > change
> > >> > > > > > > > > > > > > the API spec. The "allowed" list still makes
> > >> sense.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > In general, I like the direction this feature
> is
> > >> > taking,
> > >> > > > > > > > although I
> > >> > > > > > > > > > > > expect
> > >> > > > > > > > > > > > > PR reviews may take a few rounds :)
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Cheers,
> > >> > > > > > > > > > > > > Dmitri.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:38 PM Austen Tomek <
> > >> > > > > > > > > > > > > Austen.Tomek@chicagotrading.com> wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Hi all,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Closing the loop since my last note and
> > >> answering
> > >> > the
> > >> > > > > > > questions
> > >> > > > > > > > > > that
> > >> > > > > > > > > > > > have
> > >> > > > > > > > > > > > > > come up.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Dmitri's question on role ARNs: R2 doesn’t
> > have
> > >> > > > AWS-style
> > >> > > > > > IAM
> > >> > > > > > > > > roles
> > >> > > > > > > > > > > or
> > >> > > > > > > > > > > > > > ARNs. Our implementation signs temporary
> > >> > credentials
> > >> > > > > with a
> > >> > > > > > > > > parent
> > >> > > > > > > > > > > > > token’s
> > >> > > > > > > > > > > > > > secret, scoped to a bucket and a set of
> > >> prefixes.
> > >> > The
> > >> > > > > > account
> > >> > > > > > > > ID
> > >> > > > > > > > > > and
> > >> > > > > > > > > > > > > > jurisdiction come from the R2 endpoint.
> > Putting
> > >> the
> > >> > > > > account
> > >> > > > > > > ID
> > >> > > > > > > > in
> > >> > > > > > > > > > > > roleArn
> > >> > > > > > > > > > > > > > or userArn would give those fields a
> different
> > >> > meaning,
> > >> > > > > so
> > >> > > > > > > I’ve
> > >> > > > > > > > > > left
> > >> > > > > > > > > > > > them
> > >> > > > > > > > > > > > > > absent for CLOUDFLARE_R2.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > I pursued Option B. The S3 configuration
> gains
> > >> one
> > >> > new
> > >> > > > > > field,
> > >> > > > > > > > > > > > > > credentialIssuer, with values STS
> > >> > > > > > > > > > > > > > (default) and CLOUDFLARE_R2 and a realm
> > >> allowlist
> > >> > > > > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_ISSUERS that
> defaults
> > >> > > > > > > > > > > > > > to [STS]. I left stsUnavailable untouched
> > >> > (Prithvi) so
> > >> > > > it
> > >> > > > > > > still
> > >> > > > > > > > > > means
> > >> > > > > > > > > > > > > > "vend nothing". The
> > >> > > > > > > > > > > > > > credentialIssuer is the explicit sign. The
> R2
> > >> > config
> > >> > > > > > rejects
> > >> > > > > > > > > > > > > > stsUnavailable along with a host of other
> STS
> > >> and
> > >> > KMS
> > >> > > > > > fields.
> > >> > > > > > > > I'm
> > >> > > > > > > > > > > > running
> > >> > > > > > > > > > > > > > this in our non-production environment at
> the
> > >> > moment.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Draft PR for stage 1 (config field, API,
> > >> allowlist,
> > >> > > > > gates):
> > >> > > > > > > > > > > > > > https://github.com/apache/polaris/pull/5513
> .
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > I still see the appeal of a separate R2
> > >> > configuration,
> > >> > > > as
> > >> > > > > > > > Dmitri
> > >> > > > > > > > > > > > > > suggested. My original reason for separating
> > it
> > >> > was to
> > >> > > > > keep
> > >> > > > > > > the
> > >> > > > > > > > > > > > > R2-specific
> > >> > > > > > > > > > > > > > configuration and validation together.
> Having
> > >> > > > > > > > > > getCloudflareR2Endpoint
> > >> > > > > > > > > > > > on
> > >> > > > > > > > > > > > > > AwsStorageConfigurationInfo is one part I’d
> > >> still
> > >> > like
> > >> > > > > > > feedback
> > >> > > > > > > > > on.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > I wanted to put a working version of Option
> B
> > in
> > >> > front
> > >> > > > of
> > >> > > > > > > > > everyone
> > >> > > > > > > > > > so
> > >> > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > could discuss the actual changes. In my
> > earlier
> > >> > > > > > > implementation,
> > >> > > > > > > > > the
> > >> > > > > > > > > > > > > > separate config also meant a separate
> storage
> > >> type
> > >> > > > > sharing
> > >> > > > > > > > > > S3FileIO.
> > >> > > > > > > > > > > > That
> > >> > > > > > > > > > > > > > was the tradeoff that led me to try Option
> B.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Do the diffs change anyone’s lean here? And
> > are
> > >> the
> > >> > > > names
> > >> > > > > > > > > > > > > credentialIssuer
> > >> > > > > > > > > > > > > > / STS / CLOUDFLARE_R2 /
> > >> > SUPPORTED_S3_CREDENTIAL_ISSUERS
> > >> > > > > > > > > acceptable?
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > > Austen
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Get Outlook for Mac<
> > >> > https://aka.ms/GetOutlookForMac>
> > >> > > > > > > > > > > > > > From: Dmitri Bourlatchkov <dimas@apache.org
> >
> > >> > > > > > > > > > > > > > Date: Sunday, September 13, 2026 at 19:50
> > >> > > > > > > > > > > > > > To: dev@polaris.apache.org <
> > >> dev@polaris.apache.org
> > >> > >
> > >> > > > > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support
> > >> with
> > >> > > > scoped
> > >> > > > > > > > > credential
> > >> > > > > > > > > > > > > vending
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Building on top of Ayush's and JB's
> > suggestion,
> > >> I
> > >> > agree
> > >> > > > > > that
> > >> > > > > > > > > > > > jurisdiction
> > >> > > > > > > > > > > > > > can live inside the existing endpoint config
> > >> field.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > That leaves only Account ID, so potentially
> we
> > >> > could
> > >> > > > > store
> > >> > > > > > it
> > >> > > > > > > > in
> > >> > > > > > > > > > user
> > >> > > > > > > > > > > > or
> > >> > > > > > > > > > > > > > role ARN. Note that userArn is currently
> > unused
> > >> in
> > >> > OSS
> > >> > > > > > code.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > That should allow us to keep existing
> > >> > > > > > > > AwsStorageConfigurationInfo
> > >> > > > > > > > > > for
> > >> > > > > > > > > > > > all
> > >> > > > > > > > > > > > > > S3 storage backend, but will require special
> > >> > parsing of
> > >> > > > > > > > user/role
> > >> > > > > > > > > > ARN
> > >> > > > > > > > > > > > > field
> > >> > > > > > > > > > > > > > to distinguish R2 from the other systems.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > I'm still leaning towards the separate
> > >> > > > > > > > > R2StorageConfigurationInfo,
> > >> > > > > > > > > > > but
> > >> > > > > > > > > > > > > I'd
> > >> > > > > > > > > > > > > > be ok with overloading user/role ARN too if
> > >> other
> > >> > > > people
> > >> > > > > > > prefer
> > >> > > > > > > > > > > that. I
> > >> > > > > > > > > > > > > > guess a key question in this context is
> > whether
> > >> R2
> > >> > has
> > >> > > > > the
> > >> > > > > > > > > concept
> > >> > > > > > > > > > of
> > >> > > > > > > > > > > > > role
> > >> > > > > > > > > > > > > > ARN at all.... but TBH I have no idea about
> > >> that. I
> > >> > > > hope
> > >> > > > > > > Austen
> > >> > > > > > > > > > might
> > >> > > > > > > > > > > > be
> > >> > > > > > > > > > > > > > able to clarify that.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Cheers,
> > >> > > > > > > > > > > > > > Dmitri.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > On Sun, Sep 13, 2026 at 9:18 AM Prithvi S <
> > >> > > > > > > > > > > prithvisivasankar@gmail.com
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Hi Austen,
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Welcome, and thanks for starting with a
> > >> > [DISCUSS].
> > >> > > > Best
> > >> > > > > > way
> > >> > > > > > > > to
> > >> > > > > > > > > > > make a
> > >> > > > > > > > > > > > > > first
> > >> > > > > > > > > > > > > > > contribution :)
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > I think the thread is talking about two
> > >> layers,
> > >> > and
> > >> > > > we
> > >> > > > > > can
> > >> > > > > > > > keep
> > >> > > > > > > > > > > both
> > >> > > > > > > > > > > > > > > answers.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > On the client side this should stay S3:
> > s3://
> > >> > > > > locations,
> > >> > > > > > > s3.*
> > >> > > > > > > > > > > > > properties,
> > >> > > > > > > > > > > > > > > S3FileIO. Your PyIceberg / DuckDB /
> Iceberg
> > >> Java
> > >> > > > > results
> > >> > > > > > > > > already
> > >> > > > > > > > > > > show
> > >> > > > > > > > > > > > > > that
> > >> > > > > > > > > > > > > > > unmodified clients work, so a dedicated
> > >> R2FileIO
> > >> > does
> > >> > > > > not
> > >> > > > > > > > seem
> > >> > > > > > > > > > > worth
> > >> > > > > > > > > > > > > it.
> > >> > > > > > > > > > > > > > > On the server side I lean toward Dmitri: a
> > >> > distinct
> > >> > > > > > > > > > > > > > > R2StorageConfigurationInfo and storage
> > >> > integration.
> > >> > > > > > > Issuance
> > >> > > > > > > > is
> > >> > > > > > > > > > not
> > >> > > > > > > > > > > > > STS,
> > >> > > > > > > > > > > > > > > and folding Cloudflare fields into
> > >> > > > > > > > AwsStorageConfigurationInfo
> > >> > > > > > > > > > > mixes
> > >> > > > > > > > > > > > > > > unrelated concepts. The middle ground to
> > avoid
> > >> > is a
> > >> > > > new
> > >> > > > > > > type
> > >> > > > > > > > > that
> > >> > > > > > > > > > > is
> > >> > > > > > > > > > > > > > still
> > >> > > > > > > > > > > > > > > AWS-shaped underneath.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > One concrete caution if the Option B
> > prototype
> > >> > stays
> > >> > > > on
> > >> > > > > > the
> > >> > > > > > > > S3
> > >> > > > > > > > > > > > config:
> > >> > > > > > > > > > > > > > > please do not overload stsUnavailable.
> Today
> > >> that
> > >> > > > flag
> > >> > > > > > > skips
> > >> > > > > > > > > > > > AssumeRole
> > >> > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > disables vending. R2 wants the opposite.
> > vend,
> > >> > just
> > >> > > > not
> > >> > > > > > via
> > >> > > > > > > > > STS.
> > >> > > > > > > > > > so
> > >> > > > > > > > > > > > > that
> > >> > > > > > > > > > > > > > > flag would change meaning for existing
> > MinIO /
> > >> > NetApp
> > >> > > > > > > > catalogs.
> > >> > > > > > > > > > If
> > >> > > > > > > > > > > we
> > >> > > > > > > > > > > > > > stay
> > >> > > > > > > > > > > > > > > on S3 config, we need an explicit vending
> > >> > signal; a
> > >> > > > > > > dedicated
> > >> > > > > > > > > R2
> > >> > > > > > > > > > > type
> > >> > > > > > > > > > > > > > > avoids that trap.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > On config surface, jurisdiction can live
> in
> > >> > endpoint
> > >> > > > > > > > > (<account>.
> > >> > > > > > > > > > > > > > > r2.cloudflarestorage.com vs
> > >> <account>.eu.r2...).
> > >> > > > > Account
> > >> > > > > > > ID
> > >> > > > > > > > is
> > >> > > > > > > > > > > still
> > >> > > > > > > > > > > > > > > needed
> > >> > > > > > > > > > > > > > > for the JWT sub, so keeping an explicit
> > >> > accountId and
> > >> > > > > > > > deriving
> > >> > > > > > > > > > > > audience
> > >> > > > > > > > > > > > > > > from the endpoint seems enough. Parent
> > tokens
> > >> > should
> > >> > > > > stay
> > >> > > > > > > in
> > >> > > > > > > > > > server
> > >> > > > > > > > > > > > > > config
> > >> > > > > > > > > > > > > > > via storageName.
> > >> > > > > > > > > > > > > > > Local signing fits the vending model.
> > >> > Fail-closed on
> > >> > > > > > > > > cross-bucket
> > >> > > > > > > > > > > and
> > >> > > > > > > > > > > > > > mixed
> > >> > > > > > > > > > > > > > > read/write grants is the right call.
> Prefix
> > >> > claims
> > >> > > > > should
> > >> > > > > > > > > follow
> > >> > > > > > > > > > > the
> > >> > > > > > > > > > > > > same
> > >> > > > > > > > > > > > > > > path comparison Polaris already uses, so
> we
> > do
> > >> > not
> > >> > > > > mint a
> > >> > > > > > > > > > > credential
> > >> > > > > > > > > > > > > > wider
> > >> > > > > > > > > > > > > > > than the grant.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > +1 to small PRs: FileIO discriminator
> > (#4486)
> > >> > > > > separately
> > >> > > > > > if
> > >> > > > > > > > R2
> > >> > > > > > > > > > > needs
> > >> > > > > > > > > > > > > it,
> > >> > > > > > > > > > > > > > > then management API + Java config, then
> > >> vending,
> > >> > > > Python
> > >> > > > > > CLI
> > >> > > > > > > > on
> > >> > > > > > > > > > its
> > >> > > > > > > > > > > > own.
> > >> > > > > > > > > > > > > > > Sketching MinIO / Backblaze against the
> same
> > >> > fields
> > >> > > > is
> > >> > > > > a
> > >> > > > > > > > useful
> > >> > > > > > > > > > > > design
> > >> > > > > > > > > > > > > > > check; a generic non-STS SPI can wait for
> a
> > >> > second
> > >> > > > > > backend.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Regards,
> > >> > > > > > > > > > > > > > > Prithvi S
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > On Fri, Sep 11, 2026 at 4:31 AM Dmitri
> > >> > Bourlatchkov <
> > >> > > > > > > > > > > > dimas@apache.org>
> > >> > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Hi Austen,
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Thanks for starting this contribution!
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > After briefly looking through your PR
> [1]
> > I
> > >> > think
> > >> > > > > > > > > > > > > > > > adding R2StorageConfigInfo makes sense.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Even though "aws" in
> > >> > AwsStorageConfigurationInfo is
> > >> > > > > > > longer
> > >> > > > > > > > > > > implies
> > >> > > > > > > > > > > > > AWS
> > >> > > > > > > > > > > > > > > > services, and MinIO, Rust, Ozone are
> > >> supported
> > >> > just
> > >> > > > > as
> > >> > > > > > > > well,
> > >> > > > > > > > > R2
> > >> > > > > > > > > > > > > appears
> > >> > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > be significantly different from
> STS-based
> > S3
> > >> > > > systems
> > >> > > > > > > that a
> > >> > > > > > > > > > > > separate
> > >> > > > > > > > > > > > > > > config
> > >> > > > > > > > > > > > > > > > is probably the most natural way to
> > support
> > >> it
> > >> > in
> > >> > > > the
> > >> > > > > > > > Polaris
> > >> > > > > > > > > > > > > codebase.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > The alternative, I guess, is to add R2
> > >> > accountId
> > >> > > > and
> > >> > > > > > > > > > jurisdiction
> > >> > > > > > > > > > > > as
> > >> > > > > > > > > > > > > > > > optional properties to
> > >> > AwsStorageConfigurationInfo
> > >> > > > > (or
> > >> > > > > > > > > overload
> > >> > > > > > > > > > > > > > existing
> > >> > > > > > > > > > > > > > > > properties), but that looks like mixing
> > >> > unrelated
> > >> > > > > > > concepts
> > >> > > > > > > > > > > > together.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Adding a generic property bag to
> > >> > > > > > > > AwsStorageConfigurationInfo
> > >> > > > > > > > > is
> > >> > > > > > > > > > > not
> > >> > > > > > > > > > > > > > > > convenient because
> > >> > > > > > PolarisStorageIntegrationProviderImpl
> > >> > > > > > > > will
> > >> > > > > > > > > > > > > probably
> > >> > > > > > > > > > > > > > > have
> > >> > > > > > > > > > > > > > > > to know how to interpret it in order to
> > >> > redirect
> > >> > > > > > > credential
> > >> > > > > > > > > > > vending
> > >> > > > > > > > > > > > > > > > requests to R2-specific code.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > I wonder whether
> > R2StorageConfigurationInfo
> > >> and
> > >> > > > > > > > > > > > > > > AwsStorageConfigurationInfo
> > >> > > > > > > > > > > > > > > > (and corresponding "storage integration"
> > >> > classes)
> > >> > > > > could
> > >> > > > > > > > > share a
> > >> > > > > > > > > > > > > common
> > >> > > > > > > > > > > > > > > base
> > >> > > > > > > > > > > > > > > > class for common properties (e.g.
> > endpoint).
> > >> > Even
> > >> > > > > > though
> > >> > > > > > > > > > > "endpoint"
> > >> > > > > > > > > > > > > > might
> > >> > > > > > > > > > > > > > > > not be relevant to R2 in the cloud, the
> > fact
> > >> > that
> > >> > > > > > > S3FileIO
> > >> > > > > > > > > uses
> > >> > > > > > > > > > > it,
> > >> > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > R2
> > >> > > > > > > > > > > > > > > > is accessed via S3FileIO means we should
> > >> > probably
> > >> > > > > allow
> > >> > > > > > > > users
> > >> > > > > > > > > > to
> > >> > > > > > > > > > > > > > > configure
> > >> > > > > > > > > > > > > > > > the full set of FileIO properties just
> in
> > >> case.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > I think it is fine for different
> > >> > > > > > > PolarisStorageIntegration
> > >> > > > > > > > > > > > > > > implementations
> > >> > > > > > > > > > > > > > > > to produce the same kind of
> > >> > StorageAccessConfig (S3
> > >> > > > > in
> > >> > > > > > > this
> > >> > > > > > > > > > case)
> > >> > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > thus
> > >> > > > > > > > > > > > > > > > cause S3FileIO to be used by clients
> > >> (including
> > >> > > > > > Polaris'
> > >> > > > > > > > own
> > >> > > > > > > > > > > > storage
> > >> > > > > > > > > > > > > > > access
> > >> > > > > > > > > > > > > > > > paths).
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Changes in StorageTypeFileIO in [1]
> look a
> > >> bit
> > >> > > > > > concerning
> > >> > > > > > > > to
> > >> > > > > > > > > > me.
> > >> > > > > > > > > > > I
> > >> > > > > > > > > > > > > > wonder
> > >> > > > > > > > > > > > > > > > if this code could be refactored to
> avoid
> > >> > > > > dependencies
> > >> > > > > > on
> > >> > > > > > > > > > storage
> > >> > > > > > > > > > > > > > config
> > >> > > > > > > > > > > > > > > > types completely.... but TBH, I did not
> > look
> > >> > too
> > >> > > > > deeply
> > >> > > > > > > > into
> > >> > > > > > > > > > this
> > >> > > > > > > > > > > > > > today.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Anticipating future PRs for this in the
> > >> Polaris
> > >> > > > repo,
> > >> > > > > > I'd
> > >> > > > > > > > > like
> > >> > > > > > > > > > to
> > >> > > > > > > > > > > > ask
> > >> > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > separate Python code changes from java
> > >> changes
> > >> > > > since
> > >> > > > > > > these
> > >> > > > > > > > > > areas
> > >> > > > > > > > > > > > > > usually
> > >> > > > > > > > > > > > > > > > attract different reviewers :)
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > From my POV, management API and matching
> > >> java
> > >> > > > changes
> > >> > > > > > can
> > >> > > > > > > > be
> > >> > > > > > > > > in
> > >> > > > > > > > > > > the
> > >> > > > > > > > > > > > > > same
> > >> > > > > > > > > > > > > > > > PR. I'd say docs should come later (to
> > keep
> > >> PRs
> > >> > > > > small).
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > All in all, thank you again and I hope
> > this
> > >> > > > > > contribution
> > >> > > > > > > > > lands
> > >> > > > > > > > > > in
> > >> > > > > > > > > > > > > > Polaris
> > >> > > > > > > > > > > > > > > > :)
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > [1]
> > >> > https://github.com/deepdishgary/polaris/pull/1
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Cheers,
> > >> > > > > > > > > > > > > > > > Dmitri.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > On Thu, Sep 10, 2026 at 1:52 PM
> > >> Jean-Baptiste
> > >> > > > Onofré
> > >> > > > > <
> > >> > > > > > > > > > > > > jb@nanthrax.net>
> > >> > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Hi Austen,
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Nice discussion! Thanks for that, and
> > >> welcome
> > >> > > > > aboard
> > >> > > > > > :)
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > I think it makes sense to have R2
> built
> > >> > under the
> > >> > > > > > > > existing
> > >> > > > > > > > > S3
> > >> > > > > > > > > > > > > storage
> > >> > > > > > > > > > > > > > > > > configuration. It matches how Polaris
> is
> > >> > already
> > >> > > > > > > > > structured:
> > >> > > > > > > > > > > > > > > > > StorageType is keyed on the URI scheme
> > >> > (s3://,
> > >> > > > > > s3a://),
> > >> > > > > > > > and
> > >> > > > > > > > > > > > > > > > > AwsStorageConfigurationInfo already
> > >> carries
> > >> > > > > endpoint,
> > >> > > > > > > > > > > > stsEndpoint,
> > >> > > > > > > > > > > > > > > > > pathStyleAccess and, importantly, an
> > >> > > > stsUnavailable
> > >> > > > > > > flag.
> > >> > > > > > > > > > > > > > > > > So, what you need is largely there
> > >> already:
> > >> > > > correct
> > >> > > > > > me
> > >> > > > > > > if
> > >> > > > > > > > > I'm
> > >> > > > > > > > > > > > > wrong,
> > >> > > > > > > > > > > > > > > > > R2 is essentially "S3 compatible store
> > >> where
> > >> > STS
> > >> > > > is
> > >> > > > > > > > > > > unavailable,
> > >> > > > > > > > > > > > > plus
> > >> > > > > > > > > > > > > > > > > a way to actually vend credentials in
> > that
> > >> > case".
> > >> > > > > > Going
> > >> > > > > > > > > this
> > >> > > > > > > > > > > > route
> > >> > > > > > > > > > > > > > > > > also keeps this work "isolated" from
> the
> > >> > > > > > > > > > > FileIO-per-storage-type
> > >> > > > > > > > > > > > > > > > > discussion.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > The interesting design question is a
> > >> > pluggable
> > >> > > > > > > credential
> > >> > > > > > > > > > > vending
> > >> > > > > > > > > > > > > > path
> > >> > > > > > > > > > > > > > > > > for S3 compatible stores when STS is
> not
> > >> > > > available,
> > >> > > > > > > with
> > >> > > > > > > > > > > > > Cloudflare's
> > >> > > > > > > > > > > > > > > > > local signing as the first concrete
> > >> > > > implementation,
> > >> > > > > > > > rather
> > >> > > > > > > > > > than
> > >> > > > > > > > > > > > R2
> > >> > > > > > > > > > > > > > > > > specific code. It would be good to
> > sketch
> > >> how
> > >> > > > MinIO
> > >> > > > > > > would
> > >> > > > > > > > > > slot
> > >> > > > > > > > > > > > into
> > >> > > > > > > > > > > > > > > > > the same config fields before the
> > vending
> > >> > path
> > >> > > > > > hardens
> > >> > > > > > > > > around
> > >> > > > > > > > > > > R2.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > I believe we have to check if
> Cloudflare
> > >> > account
> > >> > > > ID
> > >> > > > > > and
> > >> > > > > > > > > > > > > jurisdiction
> > >> > > > > > > > > > > > > > > > > really need dedicated config fields,
> or
> > >> can
> > >> > we
> > >> > > > use
> > >> > > > > > the
> > >> > > > > > > > > > endpoint
> > >> > > > > > > > > > > > > value
> > >> > > > > > > > > > > > > > > > > (with documentation)? Keeping the
> > >> > configuration
> > >> > > > > > surface
> > >> > > > > > > > > > minimal
> > >> > > > > > > > > > > > > will
> > >> > > > > > > > > > > > > > > > > help the design generalize.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > I would love to discuss two things:
> > >> > > > > > > > > > > > > > > > > 1. Polaris shipping a Cloudflare API
> > token
> > >> > and
> > >> > > > > > locally
> > >> > > > > > > > > > minting
> > >> > > > > > > > > > > > > scoped
> > >> > > > > > > > > > > > > > > > > credentials makes it effectively an
> STS
> > >> for
> > >> > R2,
> > >> > > > > which
> > >> > > > > > > > > > increases
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > blast radius of a server compromise.
> > Your
> > >> > "reject
> > >> > > > > > > rather
> > >> > > > > > > > > than
> > >> > > > > > > > > > > > > > boarden"
> > >> > > > > > > > > > > > > > > > > handling of unrepresentable scopes is
> > the
> > >> > right
> > >> > > > > call.
> > >> > > > > > > > > > > > > > > > > 2. The PyIceberg/DuckDB/Java results
> are
> > >> > > > promising.
> > >> > > > > > > Just
> > >> > > > > > > > > > > curious
> > >> > > > > > > > > > > > > > about
> > >> > > > > > > > > > > > > > > > > Spark or Trino.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > We love small, well-scoped PRs. I
> would
> > >> > suggest
> > >> > > > to
> > >> > > > > > > start
> > >> > > > > > > > > with
> > >> > > > > > > > > > > the
> > >> > > > > > > > > > > > > S3
> > >> > > > > > > > > > > > > > > > > config and the management API as first
> > >> > shoot, and
> > >> > > > > we
> > >> > > > > > > can
> > >> > > > > > > > > > follow
> > >> > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > the credential vending implementation.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Regards
> > >> > > > > > > > > > > > > > > > > JB
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > On Wed, Sep 9, 2026 at 8:18 PM Austen
> > >> Tomek
> > >> > > > > > > > > > > > > > > > > <Au...@chicagotrading.com>
> > wrote:
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Hi all,
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > I’m Austen Tomek from Chicago
> Trading
> > >> > Company.
> > >> > > > We
> > >> > > > > > use
> > >> > > > > > > > > > Polaris
> > >> > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > our
> > >> > > > > > > > > > > > > > > > > Iceberg catalogs and we're evaluating
> > >> > Cloudflare
> > >> > > > R2
> > >> > > > > > as
> > >> > > > > > > an
> > >> > > > > > > > > > > object
> > >> > > > > > > > > > > > > > store
> > >> > > > > > > > > > > > > > > > (the
> > >> > > > > > > > > > > > > > > > > no egress fees make it a very
> compelling
> > >> > > > product).
> > >> > > > > > I’d
> > >> > > > > > > > like
> > >> > > > > > > > > > to
> > >> > > > > > > > > > > > > > discuss
> > >> > > > > > > > > > > > > > > > > contributing R2 support and credential
> > >> > vending to
> > >> > > > > > > Polaris
> > >> > > > > > > > > > with
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > goal
> > >> > > > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > getting feedback before opening a
> > >> > > > ready-for-review
> > >> > > > > > > > upstream
> > >> > > > > > > > > > PR.
> > >> > > > > > > > > > > > > This
> > >> > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > my
> > >> > > > > > > > > > > > > > > > > first contribution, so trying to do
> this
> > >> > right.
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > R2 exposes S3-compatible APIs but it
> > >> does
> > >> > not
> > >> > > > > > provide
> > >> > > > > > > > AWS
> > >> > > > > > > > > > > STS.
> > >> > > > > > > > > > > > > > > > > Cloudflare instead supports these
> > >> > short-lived,
> > >> > > > > scoped
> > >> > > > > > > > > > > credentials
> > >> > > > > > > > > > > > > > > > generated
> > >> > > > > > > > > > > > > > > > > by locally signed JWT with the server
> > >> > holding the
> > >> > > > > > > parent
> > >> > > > > > > > > API
> > >> > > > > > > > > > > > token
> > >> > > > > > > > > > > > > > [1].
> > >> > > > > > > > > > > > > > > > > This allows us to retain Polaris’s
> > >> > authorization
> > >> > > > > and
> > >> > > > > > > > > > > > > > credential-vending
> > >> > > > > > > > > > > > > > > > > model while storing our table data in
> > R2.
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > I have a prototype with the
> following
> > >> > design:
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >   *
> > >> > > > > > > > > > > > > > > > > > An opt-in R2 storage type, with
> > catalog
> > >> > > > > > configuration
> > >> > > > > > > > > > > > identifying
> > >> > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > Cloudflare account and optional
> > >> jurisdiction.
> > >> > > > > Parent
> > >> > > > > > > > > > > credentials
> > >> > > > > > > > > > > > > > remain
> > >> > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > server configuration and can be
> selected
> > >> > through
> > >> > > > > > > > > storageName.
> > >> > > > > > > > > > > > > > > > > >   *
> > >> > > > > > > > > > > > > > > > > > Local credential generation using
> > >> > Cloudflare’s
> > >> > > > > > > > documented
> > >> > > > > > > > > > > > format.
> > >> > > > > > > > > > > > > > > > > Credentials expire, are restricted to
> > one
> > >> > bucket,
> > >> > > > > and
> > >> > > > > > > > carry
> > >> > > > > > > > > > > > prefix
> > >> > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > access scopes derived from Polaris’s
> > >> location
> > >> > > > > grants.
> > >> > > > > > > The
> > >> > > > > > > > > > > > > integration
> > >> > > > > > > > > > > > > > > > uses
> > >> > > > > > > > > > > > > > > > > Polaris’s existing credential cache.
> > >> > > > > > > > > > > > > > > > > >   *
> > >> > > > > > > > > > > > > > > > > > Credentials returned through the
> > >> existing
> > >> > > > Iceberg
> > >> > > > > > > REST
> > >> > > > > > > > > > > vending
> > >> > > > > > > > > > > > > flow
> > >> > > > > > > > > > > > > > > as
> > >> > > > > > > > > > > > > > > > > s3.* properties, including the
> endpoint,
> > >> > session
> > >> > > > > > token,
> > >> > > > > > > > and
> > >> > > > > > > > > > > > expiry.
> > >> > > > > > > > > > > > > > > > > Compatible clients use their existing
> S3
> > >> > FileIO.
> > >> > > > > > > > > > > > > > > > > >   *
> > >> > > > > > > > > > > > > > > > > > Conservative handling of scopes the
> > >> > > > > implementation
> > >> > > > > > > > cannot
> > >> > > > > > > > > > > > > > represent:
> > >> > > > > > > > > > > > > > > > > cross-bucket and mixed read/write
> grants
> > >> are
> > >> > > > > rejected
> > >> > > > > > > > > rather
> > >> > > > > > > > > > > than
> > >> > > > > > > > > > > > > > > > > broadened. R2 is excluded from the
> > default
> > >> > > > > > > > > > > > supported-storage-types
> > >> > > > > > > > > > > > > > > list.
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > The main design questions I feel
> this
> > >> runs
> > >> > > > into:
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >   *
> > >> > > > > > > > > > > > > > > > > > Should this be a distinct storage
> type
> > >> or a
> > >> > > > > > > > > > > credential-vending
> > >> > > > > > > > > > > > > > > > mechanism
> > >> > > > > > > > > > > > > > > > > selected through the existing S3
> > >> > configuration?
> > >> > > > > > > > > > > > > > > > > >      *
> > >> > > > > > > > > > > > > > > > > > I chose a separate type because the
> > >> > identity,
> > >> > > > > > > > credential
> > >> > > > > > > > > > > > > > generation,
> > >> > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > scoping model differ from AWS STS, as
> > this
> > >> > seemed
> > >> > > > > > more
> > >> > > > > > > > > > > > appropriate
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > There is a draft preview PR against
> my
> > >> fork
> > >> > > > [2],
> > >> > > > > > > > > including
> > >> > > > > > > > > > > > > > > > > implementation, management API
> changes,
> > >> CLI
> > >> > > > > support,
> > >> > > > > > > > tests,
> > >> > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > documentation. It also discloses the
> AI
> > >> > > > assistance
> > >> > > > > > used
> > >> > > > > > > > > > during
> > >> > > > > > > > > > > > > > > > > implementation. I leaned heavily on
> > Fable
> > >> > 5.1 for
> > >> > > > > > this
> > >> > > > > > > > > > > > > implementation
> > >> > > > > > > > > > > > > > > so
> > >> > > > > > > > > > > > > > > > I
> > >> > > > > > > > > > > > > > > > > want to be upfront about it.
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > We are running the implementation
> in a
> > >> > > > > > non-production
> > >> > > > > > > > > > > > environment
> > >> > > > > > > > > > > > > > > right
> > >> > > > > > > > > > > > > > > > > now as a POC. Validation includes live
> > R2
> > >> > > > > > reads/writes,
> > >> > > > > > > > > > > multipart
> > >> > > > > > > > > > > > > > > > > operations, negative scope-boundary
> > tests,
> > >> > our
> > >> > > > > > internal
> > >> > > > > > > > > > Python
> > >> > > > > > > > > > > > > client
> > >> > > > > > > > > > > > > > > > > regression matrix, and credential
> > >> > > > rotation/recovery
> > >> > > > > > > > > > exercises.
> > >> > > > > > > > > > > > The
> > >> > > > > > > > > > > > > > > > upstream
> > >> > > > > > > > > > > > > > > > > CI workflow has passed on the fork.
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > The change also encounters the
> > existing
> > >> > > > > assumption
> > >> > > > > > > > that a
> > >> > > > > > > > > > > > FileIO
> > >> > > > > > > > > > > > > > > > > implementation identifies exactly one
> > >> storage
> > >> > > > type:
> > >> > > > > > > both
> > >> > > > > > > > S3
> > >> > > > > > > > > > and
> > >> > > > > > > > > > > > R2
> > >> > > > > > > > > > > > > > use
> > >> > > > > > > > > > > > > > > > > S3FileIO. This overlaps with #4486
> [3].
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > I’d particularly appreciate feedback
> > on:
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >   1.
> > >> > > > > > > > > > > > > > > > > > Should R2 have its own storage
> > >> > > > > configuration/type,
> > >> > > > > > or
> > >> > > > > > > > > > should
> > >> > > > > > > > > > > > > > non-STS
> > >> > > > > > > > > > > > > > > > > vending be modeled within the existing
> > S3
> > >> > > > > > integration?
> > >> > > > > > > > > > > > > > > > > >   2.
> > >> > > > > > > > > > > > > > > > > > Does the local-signing approach fit
> > >> > Polaris’s
> > >> > > > > > > > > > > > credential-vending
> > >> > > > > > > > > > > > > > > model?
> > >> > > > > > > > > > > > > > > > > Any additional constraints or
> > validations
> > >> > > > > necessary?
> > >> > > > > > > > > > > > > > > > > >   3.
> > >> > > > > > > > > > > > > > > > > > Would you prefer the shared FileIO
> > >> > validation
> > >> > > > > > change
> > >> > > > > > > to
> > >> > > > > > > > > > land
> > >> > > > > > > > > > > > > > > > separately,
> > >> > > > > > > > > > > > > > > > > and how should we sequence the
> > management
> > >> > API and
> > >> > > > > R2
> > >> > > > > > > > > > > integration
> > >> > > > > > > > > > > > > > > changes?
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >   Happy to adapt the design and
> split
> > >> the
> > >> > > > > > > contribution
> > >> > > > > > > > > into
> > >> > > > > > > > > > > > more
> > >> > > > > > > > > > > > > > > > focused
> > >> > > > > > > > > > > > > > > > > PRs. Wanted to start the discussion
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >   Thanks,
> > >> > > > > > > > > > > > > > > > > >   Austen
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >   [1]
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > >
> > >> https://developers.cloudflare.com/r2/api/s3/temporary-credentials/
> > >> > > > > > > > > > > > > > > > > >   [2]
> > >> > > > > > https://github.com/deepdishgary/polaris/pull/1
> > >> > > > > > > > > > > > > > > > > >   [3]
> > >> > > > > > https://github.com/apache/polaris/issues/4486
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Get Outlook for Mac<
> > >> > > > > > https://aka.ms/GetOutlookForMac>
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > This electronic mail message and any
> > >> > attached
> > >> > > > > files
> > >> > > > > > > > > contain
> > >> > > > > > > > > > > > > > > information
> > >> > > > > > > > > > > > > > > > > intended for the exclusive use of the
> > >> > individual
> > >> > > > or
> > >> > > > > > > > entity
> > >> > > > > > > > > to
> > >> > > > > > > > > > > > whom
> > >> > > > > > > > > > > > > it
> > >> > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > > addressed and may contain information
> > >> that is
> > >> > > > > > > > proprietary,
> > >> > > > > > > > > > > > > > confidential
> > >> > > > > > > > > > > > > > > > > and/or exempt from disclosure under
> > >> > applicable
> > >> > > > law.
> > >> > > > > > If
> > >> > > > > > > > you
> > >> > > > > > > > > > are
> > >> > > > > > > > > > > > not
> > >> > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > intended recipient, you are hereby
> > >> notified
> > >> > that
> > >> > > > > any
> > >> > > > > > > > > viewing,
> > >> > > > > > > > > > > > > > copying,
> > >> > > > > > > > > > > > > > > > > disclosure or distribution of this
> > >> > information
> > >> > > > may
> > >> > > > > be
> > >> > > > > > > > > subject
> > >> > > > > > > > > > > to
> > >> > > > > > > > > > > > > > legal
> > >> > > > > > > > > > > > > > > > > restriction or sanction. Please notify
> > the
> > >> > > > sender,
> > >> > > > > by
> > >> > > > > > > > > > > electronic
> > >> > > > > > > > > > > > > mail
> > >> > > > > > > > > > > > > > > or
> > >> > > > > > > > > > > > > > > > > telephone, of any unintended
> recipients
> > >> and
> > >> > > > delete
> > >> > > > > > the
> > >> > > > > > > > > > original
> > >> > > > > > > > > > > > > > message
> > >> > > > > > > > > > > > > > > > > without making any copies.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > This electronic mail message and any
> attached
> > >> files
> > >> > > > > contain
> > >> > > > > > > > > > > information
> > >> > > > > > > > > > > > > > intended for the exclusive use of the
> > >> individual or
> > >> > > > > entity
> > >> > > > > > to
> > >> > > > > > > > > whom
> > >> > > > > > > > > > it
> > >> > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > addressed and may contain information that
> is
> > >> > > > > proprietary,
> > >> > > > > > > > > > > confidential
> > >> > > > > > > > > > > > > > and/or exempt from disclosure under
> applicable
> > >> > law. If
> > >> > > > > you
> > >> > > > > > > are
> > >> > > > > > > > > not
> > >> > > > > > > > > > > the
> > >> > > > > > > > > > > > > > intended recipient, you are hereby notified
> > that
> > >> > any
> > >> > > > > > viewing,
> > >> > > > > > > > > > > copying,
> > >> > > > > > > > > > > > > > disclosure or distribution of this
> information
> > >> may
> > >> > be
> > >> > > > > > subject
> > >> > > > > > > > to
> > >> > > > > > > > > > > legal
> > >> > > > > > > > > > > > > > restriction or sanction. Please notify the
> > >> sender,
> > >> > by
> > >> > > > > > > > electronic
> > >> > > > > > > > > > mail
> > >> > > > > > > > > > > > or
> > >> > > > > > > > > > > > > > telephone, of any unintended recipients and
> > >> delete
> > >> > the
> > >> > > > > > > original
> > >> > > > > > > > > > > message
> > >> > > > > > > > > > > > > > without making any copies.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > This electronic mail message and any attached
> files
> > >> > contain
> > >> > > > > > > > information
> > >> > > > > > > > > > > intended for the exclusive use of the individual
> or
> > >> > entity to
> > >> > > > > > whom
> > >> > > > > > > it
> > >> > > > > > > > > is
> > >> > > > > > > > > > > addressed and may contain information that is
> > >> > proprietary,
> > >> > > > > > > > confidential
> > >> > > > > > > > > > > and/or exempt from disclosure under applicable
> law.
> > If
> > >> > you
> > >> > > > are
> > >> > > > > > not
> > >> > > > > > > > the
> > >> > > > > > > > > > > intended recipient, you are hereby notified that
> any
> > >> > viewing,
> > >> > > > > > > > copying,
> > >> > > > > > > > > > > disclosure or distribution of this information may
> > be
> > >> > subject
> > >> > > > > to
> > >> > > > > > > > legal
> > >> > > > > > > > > > > restriction or sanction. Please notify the sender,
> > by
> > >> > > > > electronic
> > >> > > > > > > mail
> > >> > > > > > > > > or
> > >> > > > > > > > > > > telephone, of any unintended recipients and delete
> > the
> > >> > > > original
> > >> > > > > > > > message
> > >> > > > > > > > > > > without making any copies.
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > This electronic mail message and any attached files
> > >> contain
> > >> > > > > > > information
> > >> > > > > > > > > > intended for the exclusive use of the individual or
> > >> entity
> > >> > to
> > >> > > > > whom
> > >> > > > > > it
> > >> > > > > > > > is
> > >> > > > > > > > > > addressed and may contain information that is
> > >> proprietary,
> > >> > > > > > > confidential
> > >> > > > > > > > > > and/or exempt from disclosure under applicable law.
> If
> > >> you
> > >> > are
> > >> > > > > not
> > >> > > > > > > the
> > >> > > > > > > > > > intended recipient, you are hereby notified that any
> > >> > viewing,
> > >> > > > > > > copying,
> > >> > > > > > > > > > disclosure or distribution of this information may
> be
> > >> > subject
> > >> > > > to
> > >> > > > > > > legal
> > >> > > > > > > > > > restriction or sanction. Please notify the sender,
> by
> > >> > > > electronic
> > >> > > > > > mail
> > >> > > > > > > > or
> > >> > > > > > > > > > telephone, of any unintended recipients and delete
> the
> > >> > original
> > >> > > > > > > message
> > >> > > > > > > > > > without making any copies.
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > This electronic mail message and any attached files
> > contain
> > >> > > > > information
> > >> > > > > > > > intended for the exclusive use of the individual or
> entity
> > >> to
> > >> > whom
> > >> > > > it
> > >> > > > > > is
> > >> > > > > > > > addressed and may contain information that is
> proprietary,
> > >> > > > > confidential
> > >> > > > > > > > and/or exempt from disclosure under applicable law. If
> you
> > >> are
> > >> > not
> > >> > > > > the
> > >> > > > > > > > intended recipient, you are hereby notified that any
> > >> viewing,
> > >> > > > > copying,
> > >> > > > > > > > disclosure or distribution of this information may be
> > >> subject
> > >> > to
> > >> > > > > legal
> > >> > > > > > > > restriction or sanction. Please notify the sender, by
> > >> > electronic
> > >> > > > mail
> > >> > > > > > or
> > >> > > > > > > > telephone, of any unintended recipients and delete the
> > >> original
> > >> > > > > message
> > >> > > > > > > > without making any copies.
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] Cloudflare R2 support with scoped credential vending

[Yufei Gu <fl...@gmail.com>]

Thanks for the summary, Dmitri!

+1 on the initial list ("STS" and "R2"). These will be written to the REST
spec along with the new key introduced (credentialVendingMechanism).

Yufei


On Thu, Oct 1, 2026 at 10:14 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> Recap from today's community sync call:
>
> * Actual set of values acceptable for "credentialVendingMechanism" is a
> finite list. Initially "STS" and "R2" will be supported by Polaris (exact
> names TBD during R2 implementation).
>
> * Servers validate values in "credentialVendingMechanism" in runtime after
> parsing the request.
>
> * The Open API doc defines "credentialVendingMechanism" as a string to
> allow for flexibility in auto-generated code. Clients / parsers do not have
> to re-build when a server adds a new option for
> "credentialVendingMechanism".
>
> Please correct me if I missed anything.
>
> Does this sound acceptable?
>
> Thanks,
> Dmitri.
>
> On Wed, Sep 30, 2026 at 1:30 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Yufei,
> >
> > Thanks for providing the details! Replying inline.
> >
> > > When a client creates a catalog with credentialVendingMechanism: "m1",
> > > what is the expected behavior?
> >
> > This depends on the meaning of "m1" and the specific server build.
> >
> > In Apache Polaris "m1" is not meaningful and is not supported.
> > The server will reject this value on the corresponding catalog
> > update request. This is handled in PolarisAdminService in the current PR
> > state.
> >
> > > 1.  The server supports it: This is the ideal scenario (happy path).
> >
> > Indeed.
> >
> > > 2.  The server does not support it: This creates inconsistency if a
> > client
> > > that works with Server A fails on Server B.
> >
> > In my view this is not an inconsistency as different servers do not have
> > to provide the same capabilities.
> >
> > Creating a catalog with a specific credentialVendingMechanism is an
> > administrative task. The admin user must be aware of what the server
> > supports. This is a deployment / configuration concern. Not a runtime
> > compatibility concern.
> >
> > > 3.  The server supports "m1" differently: This is the worst-case
> > scenario,
> > > where the client's expectations conflict with the server's actual
> > behavior.
> >
> > This is a good point!
> >
> > My take is that if "m1" is defined in Apache Polaris, then all server
> > builds must implement it the same way.
> >
> > The definition will be expressed by means of S3CredentialVendingMechanism
> > classes included in standard Apache Polaris releases (and hopefully docs
> > too).
> >
> > Currently only "STS" and "DEFAULT" (falling back on STS) are defined now.
> >
> > If "m1" is not defined in Apache Polaris, custom builds are free to
> assign
> > any meaning to it and do not have to be mutually compatible. This is the
> > main point for extensibility.
> >
> > Note that custom builds are not "Apache Polaris" anymore and cannot
> > claim to be the "same" as Apache Polaris. This is a matter of ASF
> > trademarks.
> > Normally, custom builds say something like "Powered by Apache Polaris".
> >
> > Again, the admin user setting "m1" as a vending mechanism in a particular
> > server must be aware of what that server provides and understand
> > the meaning of "m1" in that context. The onus to describe the meaning of
> > "m1" to users in this case rests with the custom server developers.
> >
> > We can talk about preventing name overlaps, but I do not think
> establishing
> > any kind of name registry is worth the trouble. In this day and age, I
> > believe
> > all downstream project developers are aware of name overlap issues.
> > and will follow normal practices to avoid name collisions.
> >
> > I hope this clears your concerns.
> >
> > Cheers,
> > Dmitri.
> >
> > On Wed, Sep 30, 2026 at 1:02 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> >> Hi Dmitri,
> >>
> >> When a client creates a catalog with credentialVendingMechanism: "m1",
> >> what
> >> is the expected behavior?
> >>
> >> Here is a suitaion if we allow a freeform string, the client might
> >> encounter three different scenarios depending on the server:
> >> 1.  The server supports it: This is the ideal scenario (happy path).
> >> 2.  The server does not support it: This creates inconsistency if a
> client
> >> that works with Server A fails on Server B.
> >> 3.  The server supports "m1" differently: This is the worst-case
> scenario,
> >> where the client's expectations conflict with the server's actual
> >> behavior.
> >>
> >> I don't think this is the situation we want users to run into. The fix
> is
> >> quite simple, using a specified values instead of a freefrom string.
> >>
> >> Yufei
> >>
> >>
> >> On Tue, Sep 29, 2026 at 9:48 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> >> wrote:
> >>
> >> > Hi all,
> >> >
> >> > Since concerns were raised, it's important to take a moment to align
> >> > rather than rushing. That's the way we drive consensus :)
> >> >
> >> > On the technical question of string vs enum for
> >> > credentialVendingMechanism, here is my take:
> >> >
> >> > 1. This field is part of the Management API for catalog
> >> > administrators. Iceberg REST clients (Spark, Trino, ...) never see or
> >> > interact with credentialVendingMechanism. They only consume standard
> >> > vended credentials (s3.* session tokens/keys) during table
> >> > load/commit. There is no risk of breaking query engine clients across
> >> > deployments.
> >> > 2. One of our primary goals when discussing S3-compatible stores (R2,
> >> > MinIO, ...) was to keep credential vending pluggable via SPI rather
> >> > than hardcoding vendor-specific paths in core. If we use a closed enum
> >> > in the OpenAPI spec, it means:
> >> >   * any new mechanism (including R2 in Phase 2 or MinIO) requires a
> spec
> >> > change.
> >> >   * any downstream deployment or plugin providing a custom mechanism
> >> > would break OpenAPI-generated client SDKs due to strict enum
> >> > validation.
> >> >   * runtime validation on the server (returning HTTP 400 with a clear
> >> > message) combined with the realm allowlist provides the right
> >> > server-side guardrail without restricting extensibility.
> >> > 3. Yufei's concerns about a clear contract and avoiding confusion is
> >> > valid. We can address contract clarity without closing the enum:
> >> >   * we keep type: string, but document the well-known/standard values
> >> > (STS, DEFAULT) explicitly in the description and examples in the
> >> > OpenAPI spec (as PR #5513 does).
> >> >   * we ensure the server validates cleanly with case-insensitivity (or
> >> > canonical normalization) so administrators get immediate, actionable
> >> > feedback on typos.
> >> >
> >> > PR #5513 does a solid job laying this foundation so Austen can move
> >> > forward with Phase 2. If we keep it as a string with well-documented
> >> > standard values and server-siide validation, I think it's the right
> >> > balance between spec clarify and pluggability.
> >> >
> >> > Thoughts?
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On Wed, Sep 30, 2026 at 12:50 AM Dmitri Bourlatchkov <
> dimas@apache.org>
> >> > wrote:
> >> > >
> >> > > Hi Yufei,
> >> > >
> >> > > > > Conversely, an enum will require defining all possible vending
> >> > > mechanisms in
> >> > > > > the Polaris API spec.
> >> > >
> >> > > > Yes. Otherwise, a client may failed in certain deployments.
> >> > >
> >> > > Could you describe this failure mode in terms of API
> request/response
> >> > flow?
> >> > >
> >> > > Thanks,
> >> > > Dmitri.
> >> > >
> >> > > On Tue, Sep 29, 2026 at 2:06 PM Yufei Gu <fl...@gmail.com>
> >> wrote:
> >> > >
> >> > > > > Conversely, an enum will require defining all possible vending
> >> > > > mechanisms in
> >> > > > the Polaris API spec.
> >> > > >
> >> > > > Yes. Otherwise, a client may failed in certain deployments.
> >> > > >
> >> > > > > This will prevent downstream projects from using custom vending
> >> > > > mechanisms.
> >> > > >
> >> > > > No. if downstream needs custom vending mechanisms, they can always
> >> > build
> >> > > > their own Polaris.
> >> > > >
> >> > > > Yufei
> >> > > >
> >> > > >
> >> > > > On Tue, Sep 29, 2026 at 11:01 AM Dmitri Bourlatchkov <
> >> dimas@apache.org
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Yufei,
> >> > > > >
> >> > > > > > Using an enum will clarify the REST spec contract.
> >> > > > >
> >> > > > > I'm not sure I understand your point. Current PR clearly
> describes
> >> > the
> >> > > > > meaning of "credentialVendingMechanism" and lists options
> >> available
> >> > > > > out-of-the-box.
> >> > > > >
> >> > > > > What does an enum achieve, that (current) runtime validation
> does
> >> > not?
> >> > > > >
> >> > > > > Conversely, an enum will require defining all possible vending
> >> > mechanisms
> >> > > > > in the Polaris API spec. This will prevent downstream projects
> >> from
> >> > using
> >> > > > > custom vending mechanisms.
> >> > > > >
> >> > > > > I believe Polaris as a project currently promotes downstream
> >> > > > > customizability.
> >> > > > >
> >> > > > > Cheers,
> >> > > > > Dmitri.
> >> > > > >
> >> > > > > On Tue, Sep 29, 2026 at 1:21 PM Yufei Gu <fl...@gmail.com>
> >> > wrote:
> >> > > > >
> >> > > > > > I don't think we have consensus on the PR, esp. a client
> should
> >> NOT
> >> > > > > depend
> >> > > > > > on a specific deployment. Using an enum will clarify the REST
> >> spec
> >> > > > > > contract.
> >> > > > > >
> >> > > > > > Yufei
> >> > > > > >
> >> > > > > >
> >> > > > > > On Tue, Sep 29, 2026 at 7:46 AM Dmitri Bourlatchkov <
> >> > dimas@apache.org>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi All,
> >> > > > > > >
> >> > > > > > > PR [5513] has been in review for quite some time. I believe
> we
> >> > have
> >> > > > > > general
> >> > > > > > > consensus on the Management API changes and java code
> changes.
> >> > > > > > >
> >> > > > > > > The only point that may still have divergent opinions is
> >> whether
> >> > to
> >> > > > use
> >> > > > > > > free text or enum for "credentialVendingMechanism". I
> >> personally
> >> > > > > support
> >> > > > > > > free text (rationale explained in previous comments). This
> >> > > > conversation
> >> > > > > > has
> >> > > > > > > been dormant for a few days, so I assume lazy consensus on
> the
> >> > > > current
> >> > > > > > > state of the PR (free text). In any case switching from free
> >> > text to
> >> > > > > enum
> >> > > > > > > is a non-breaking API change before the next release in case
> >> the
> >> > > > > > consensus
> >> > > > > > > shifts and we need to adjust. We have about one month for
> >> > followup
> >> > > > API
> >> > > > > > > changes.
> >> > > > > > >
> >> > > > > > > The java code introduced in this PR is not rigid and can
> >> change
> >> > at
> >> > > > any
> >> > > > > > time
> >> > > > > > > per our evolution guidelines [1].
> >> > > > > > >
> >> > > > > > > I propose merging PR [5513] on Sept 30 unless fresh blocking
> >> > concerns
> >> > > > > are
> >> > > > > > > raised.
> >> > > > > > >
> >> > > > > > > [1] https://polaris.apache.org/releases/1.8.0/evolution/
> >> > > > > > >
> >> > > > > > > [5513] https://github.com/apache/polaris/pull/5513
> >> > > > > > >
> >> > > > > > > Cheers,
> >> > > > > > > Dmitri.
> >> > > > > > >
> >> > > > > > > On Tue, Sep 29, 2026 at 9:30 AM Austen Tomek <
> >> > > > > > > Austen.Tomek@chicagotrading.com> wrote:
> >> > > > > > >
> >> > > > > > > > Hey all,
> >> > > > > > > >
> >> > > > > > > > Any chance I can get further reviews this week on the PR?
> >> I’d
> >> > like
> >> > > > to
> >> > > > > > get
> >> > > > > > > > moving onto Phase 2 here if there are no further issues to
> >> > address.
> >> > > > > > > >
> >> > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac>
> >> > > > > > > >
> >> > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> >> > > > > > > > Date: Thursday, September 24, 2026 at 17:33
> >> > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >> > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped
> >> > credential
> >> > > > > > > vending
> >> > > > > > > >
> >> > > > > > > > Hi Yufei,
> >> > > > > > > >
> >> > > > > > > > > The REST spec is a contract between Polaris servers and
> >> > clients
> >> > > > > [...]
> >> > > > > > > >
> >> > > > > > > > I tend to think that this statement is a bit too strong.
> The
> >> > > > > Management
> >> > > > > > > API
> >> > > > > > > > spec defines a protocol for clients and servers to
> >> communicate.
> >> > > > > > > >
> >> > > > > > > > > [...] This would help avoid a situation where a Polaris
> >> > client
> >> > > > > works
> >> > > > > > > > > only with certain deployments.
> >> > > > > > > >
> >> > > > > > > > Since Polaris has been developed as a customizable system,
> >> a
> >> > REST
> >> > > > > API
> >> > > > > > > > specification defined by Apache Polaris cannot control the
> >> > > > behaviour
> >> > > > > of
> >> > > > > > > all
> >> > > > > > > > possible downstream environments. The purpose of the spec
> >> is to
> >> > > > > ensure
> >> > > > > > > > protocol compatibility.
> >> > > > > > > >
> >> > > > > > > > Note that the value of `credentialVendingMechanism` has no
> >> > bearing
> >> > > > on
> >> > > > > > > > any logic clients need to execute. It is an opaque
> parameter
> >> > > > > controlled
> >> > > > > > > > by the Polaris administrator.
> >> > > > > > > >
> >> > > > > > > > > The new credentialVendingMechanism field in the REST
> spec
> >> > looks
> >> > > > > > > > > reasonable. My only suggestion is to use an enum instead
> >> of a
> >> > > > > string.
> >> > > > > > > > [...]
> >> > > > > > > >
> >> > > > > > > > The above points are exactly the reason why the Management
> >> API
> >> > > > cannot
> >> > > > > > > > enumerate all possible vending mechanisms.
> >> > > > > > > >
> >> > > > > > > > The same applies to storage types too. However, we do not
> >> have
> >> > any
> >> > > > > > > > practical
> >> > > > > > > > examples of 3rd party storage systems that go beyond
> >> > S3/GCS/ADLS
> >> > > > > (yet).
> >> > > > > > > >
> >> > > > > > > > Cheers,
> >> > > > > > > > Dmitri.
> >> > > > > > > >
> >> > > > > > > > On Thu, Sep 24, 2026 at 4:38 PM Yufei Gu <
> >> flyrain000@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Thanks Austen. It's fair that endpoints are not reliable
> >> > given
> >> > > > that
> >> > > > > > > some
> >> > > > > > > > > s3-compatible can use arbitrary hostnames.
> >> > > > > > > > > I also think stsUnavailable is fine to keep as-is. I
> don’t
> >> > think
> >> > > > we
> >> > > > > > > need
> >> > > > > > > > to
> >> > > > > > > > > change its scope or repurpose it.
> >> > > > > > > > >
> >> > > > > > > > > The new credentialVendingMechanism field in the REST
> spec
> >> > looks
> >> > > > > > > > reasonable.
> >> > > > > > > > > My only suggestion is to use an enum instead of a
> string.
> >> The
> >> > > > REST
> >> > > > > > spec
> >> > > > > > > > is
> >> > > > > > > > > a contract between Polaris servers and clients, so I’d
> >> > expect it
> >> > > > to
> >> > > > > > > > define
> >> > > > > > > > > the valid values and their meanings rather than leaving
> >> them
> >> > up
> >> > > > to
> >> > > > > > each
> >> > > > > > > > > deployment. This would help avoid a situation where a
> >> Polaris
> >> > > > > client
> >> > > > > > > > works
> >> > > > > > > > > only with certain deployments. Deployment-specific
> >> mechanisms
> >> > > > > > shouldn’t
> >> > > > > > > > be
> >> > > > > > > > > part of the standard REST contract; users who need a
> >> custom
> >> > > > > mechanism
> >> > > > > > > can
> >> > > > > > > > > always extend the server in their own deployment.
> >> > > > > > > > >
> >> > > > > > > > > > Either way the operator needs a per-realm switch like
> >> > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control
> >> which
> >> > realm
> >> > > > > can
> >> > > > > > > > mint
> >> > > > > > > > > R2 credentials from the operators Cloudflare account.
> >> > > > > > > > >
> >> > > > > > > > > Could you clarify why we need a separate per-realm
> switch?
> >> > Once
> >> > > > the
> >> > > > > > > > storage
> >> > > > > > > > > configuration is set at the catalog level to select a
> >> > specific
> >> > > > > > > mechanism,
> >> > > > > > > > > the catalog will have no choice but to support it. Do we
> >> need
> >> > > > > another
> >> > > > > > > > layer
> >> > > > > > > > > of gating?
> >> > > > > > > > >
> >> > > > > > > > > Yufei
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Wed, Sep 23, 2026 at 5:53 AM Austen Tomek <
> >> > > > > > > > > Austen.Tomek@chicagotrading.com> wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hey Yufei,
> >> > > > > > > > > >
> >> > > > > > > > > > Apologies for missing your question.
> >> > > > > > > > > >
> >> > > > > > > > > > I don’t think Endpoint is sufficient.
> >> > > > > > > > > >
> >> > > > > > > > > > Endpoint matching only works where the hostname
> >> identifies
> >> > the
> >> > > > > > vendor
> >> > > > > > > > > like
> >> > > > > > > > > > R2’s public hostnames that are recognizable, MinIO and
> >> > other
> >> > > > > > > > self-hosted
> >> > > > > > > > > > S3-compatible stores use arbitrary hostnames so there
> >> > would be
> >> > > > > > > nothing
> >> > > > > > > > to
> >> > > > > > > > > > match on. On top of that, if a match fails it would
> just
> >> > > > silently
> >> > > > > > > fall
> >> > > > > > > > > back
> >> > > > > > > > > > to STS.
> >> > > > > > > > > >
> >> > > > > > > > > > Right now the endpoint just says how you’ll connect
> not
> >> > how the
> >> > > > > > > > > > credentials are vended. There's endpoint and also
> >> > > > > endpointInternal
> >> > > > > > > for
> >> > > > > > > > > > server-side access, so it would be unclear which of
> the
> >> two
> >> > > > > > decides.
> >> > > > > > > > > >
> >> > > > > > > > > > For your other question, there is no second mechanism
> >> for
> >> > R2
> >> > > > > today,
> >> > > > > > > but
> >> > > > > > > > > > for other self-hosted stores the same endpoint could
> be
> >> > served
> >> > > > by
> >> > > > > > STS
> >> > > > > > > > or
> >> > > > > > > > > by
> >> > > > > > > > > > a different token mechanism chosen per catalog.
> >> > > > > > > > > >
> >> > > > > > > > > > stsUnavailable keeps it’s current behavior in the PR
> >> (that
> >> > was
> >> > > > > > > > Prithvi’s
> >> > > > > > > > > > point earlier in the thread)
> >> > > > > > > > > >
> >> > > > > > > > > > Either way the operator needs a per-realm switch like
> >> > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control
> >> which
> >> > > > realm
> >> > > > > > can
> >> > > > > > > > > mint
> >> > > > > > > > > > R2 credentials from the operators Cloudflare account.
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Austen
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac>
> >> > > > > > > > > >
> >> > > > > > > > > > From: Yufei Gu <fl...@gmail.com>
> >> > > > > > > > > > Date: Monday, September 21, 2026 at 17:53
> >> > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >> > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with
> scoped
> >> > > > > credential
> >> > > > > > > > > vending
> >> > > > > > > > > >
> >> > > > > > > > > > Hi Austen,
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks for the update. After looking at PR #5513, I
> >> think
> >> > we
> >> > > > > should
> >> > > > > > > > first
> >> > > > > > > > > > clarify whether we need a new
> credentialVendingMechanism
> >> > field
> >> > > > in
> >> > > > > > the
> >> > > > > > > > > > spec/config.
> >> > > > > > > > > >
> >> > > > > > > > > > Austen, you mentioned that the R2 account ID and
> >> > jurisdiction
> >> > > > > > already
> >> > > > > > > > > come
> >> > > > > > > > > > from the endpoint. Could we also use the existing
> field
> >> > like
> >> > > > > > > > `endpoint`,
> >> > > > > > > > > to
> >> > > > > > > > > > recognize R2 and select its credential-vending
> >> mechanism,
> >> > while
> >> > > > > > > keeping
> >> > > > > > > > > the
> >> > > > > > > > > > current stsUnavailable behavior unchanged?
> >> > > > > > > > > >
> >> > > > > > > > > > Are there cases where the endpoint isn’t enough, or
> >> where
> >> > users
> >> > > > > > need
> >> > > > > > > to
> >> > > > > > > > > > choose different vending mechanisms for the same
> >> > endpoint(e.g.,
> >> > > > > R2)
> >> > > > > > > per
> >> > > > > > > > > > catalog?
> >> > > > > > > > > >
> >> > > > > > > > > > On Wed, Sep 16, 2026 at 7:24 AM Austen Tomek <
> >> > > > > > > > > > Austen.Tomek@chicagotrading.com> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi Yufei, Dimitri,
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks, both. PR 1 is ready for review:
> >> > > > > > > > > > > https://github.com/apache/polaris/pull/5513
> >> > > > > > > > > > >
> >> > > > > > > > > > > I pushed a revision last night that takes in
> Dimitri’s
> >> > > > > comments.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >   *
> >> > > > > > > > > > > I changed from issuer to credentialVendingMechanism.
> >> It’s
> >> > > > also
> >> > > > > > now
> >> > > > > > > a
> >> > > > > > > > > > plain
> >> > > > > > > > > > > string and the realm allowlist is
> >> > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS, which
> >> still
> >> > > > > defaults
> >> > > > > > to
> >> > > > > > > > > [STS]
> >> > > > > > > > > > >   *
> >> > > > > > > > > > > Mechanisms are CDI beans found by their identifier,
> as
> >> > > > Dimitri
> >> > > > > > > > > proposed.
> >> > > > > > > > > > >   *
> >> > > > > > > > > > > The CLOUDFLARE_R2 bean and the R2 vending code will
> >> > follow in
> >> > > > > the
> >> > > > > > > > next
> >> > > > > > > > > PR
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > CI workflow is waiting approval to run.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks,
> >> > > > > > > > > > > Austen
> >> > > > > > > > > > >
> >> > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac
> >
> >> > > > > > > > > > >
> >> > > > > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> >> > > > > > > > > > > Date: Tuesday, September 15, 2026 at 20:00
> >> > > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >> > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with
> >> scoped
> >> > > > > > credential
> >> > > > > > > > > > vending
> >> > > > > > > > > > >
> >> > > > > > > > > > > Hi All,
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Having getCloudflareR2Endpoint on
> >> > > > AwsStorageConfigurationInfo
> >> > > > > > > > > > >
> >> > > > > > > > > > > Yes, this does feel awkward.
> >> > > > > > > > > > >
> >> > > > > > > > > > > From my POV pursuing the option with the
> >> > "credentialIssuers"
> >> > > > > > > property
> >> > > > > > > > > > looks
> >> > > > > > > > > > > promising. This property is fairly generic and fits
> >> > naturally
> >> > > > > > into
> >> > > > > > > > the
> >> > > > > > > > > > > config, given that we already target multiple
> >> backends.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Cheers,
> >> > > > > > > > > > > Dmitri.
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Tue, Sep 15, 2026 at 8:48 PM Yufei Gu <
> >> > > > flyrain000@gmail.com
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Hi Austen, Dmitri,
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Having getCloudflareR2Endpoint on
> >> > > > > AwsStorageConfigurationInfo
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > That does feel awkward. However,
> >> > > > AwsStorageConfigurationInfo
> >> > > > > > has
> >> > > > > > > > > > > supported
> >> > > > > > > > > > > > S3-compatible backends beyond AWS for a while, so
> >> > there is
> >> > > > > > > > precedent
> >> > > > > > > > > > > (e.g.,
> >> > > > > > > > > > > > MinIO, etc.). The underlying mismatch is that we
> >> call
> >> > the
> >> > > > > > storage
> >> > > > > > > > > type
> >> > > > > > > > > > > S3,
> >> > > > > > > > > > > > but its configuration mixes shared S3 settings
> with
> >> > > > > > AWS-specific
> >> > > > > > > > > ones:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > AwsStorageConfigurationInfo → storage type: S3
> >> > > > > > > > > > > >   ├── Common S3 settings
> >> > > > > > > > > > > >   │     endpoint, region, pathStyleAccess,
> >> > allowedLocations
> >> > > > > > > > > > > >   └── AWS-specific settings
> >> > > > > > > > > > > >         roleArn, externalId, STS, KMS
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > I’m open to separating the shared S3 settings from
> >> > > > > > > > provider-specific
> >> > > > > > > > > > > > configuration if that helps, but I don’t think we
> >> need
> >> > to
> >> > > > > > settle
> >> > > > > > > > on a
> >> > > > > > > > > > > > broader refactor now.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Let me know when the PR is ready for review. I’d
> be
> >> > happy
> >> > > > to
> >> > > > > > > take a
> >> > > > > > > > > > > closer
> >> > > > > > > > > > > > look.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Yufei
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Yufei
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:18 PM Dmitri
> Bourlatchkov
> >> <
> >> > > > > > > > > dimas@apache.org>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Hi Austen,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > +1 to adding "credentialIssues" or similar
> >> (renamed)
> >> > > > > property
> >> > > > > > > to
> >> > > > > > > > S3
> >> > > > > > > > > > > > config.
> >> > > > > > > > > > > > > In fact, I was looking exactly for something
> like
> >> > that
> >> > > > last
> >> > > > > > > week
> >> > > > > > > > > but
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > > idea did not quite condense in my mind :)
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > I posted some preliminary comments on the draft
> >> PR.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Just for community awareness, I'd like to
> propose
> >> > here
> >> > > > that
> >> > > > > > we
> >> > > > > > > > use
> >> > > > > > > > > > CDI
> >> > > > > > > > > > > > for
> >> > > > > > > > > > > > > finding actual "issuer" / "mechanism"
> >> > implementations.
> >> > > > The
> >> > > > > > > > codebase
> >> > > > > > > > > > has
> >> > > > > > > > > > > > > many examples of this. The closest match in this
> >> > case is
> >> > > > > > > > > > > > > probably PolarisStorageIntegrationProvider.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > This way the API does not have to enumerate all
> >> > possible
> >> > > > > > > values.
> >> > > > > > > > > > > > Downstream
> >> > > > > > > > > > > > > builds are free to add different implementations
> >> > without
> >> > > > > > having
> >> > > > > > > > to
> >> > > > > > > > > > > change
> >> > > > > > > > > > > > > the API spec. The "allowed" list still makes
> >> sense.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > In general, I like the direction this feature is
> >> > taking,
> >> > > > > > > > although I
> >> > > > > > > > > > > > expect
> >> > > > > > > > > > > > > PR reviews may take a few rounds :)
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Cheers,
> >> > > > > > > > > > > > > Dmitri.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:38 PM Austen Tomek <
> >> > > > > > > > > > > > > Austen.Tomek@chicagotrading.com> wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Hi all,
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Closing the loop since my last note and
> >> answering
> >> > the
> >> > > > > > > questions
> >> > > > > > > > > > that
> >> > > > > > > > > > > > have
> >> > > > > > > > > > > > > > come up.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Dmitri's question on role ARNs: R2 doesn’t
> have
> >> > > > AWS-style
> >> > > > > > IAM
> >> > > > > > > > > roles
> >> > > > > > > > > > > or
> >> > > > > > > > > > > > > > ARNs. Our implementation signs temporary
> >> > credentials
> >> > > > > with a
> >> > > > > > > > > parent
> >> > > > > > > > > > > > > token’s
> >> > > > > > > > > > > > > > secret, scoped to a bucket and a set of
> >> prefixes.
> >> > The
> >> > > > > > account
> >> > > > > > > > ID
> >> > > > > > > > > > and
> >> > > > > > > > > > > > > > jurisdiction come from the R2 endpoint.
> Putting
> >> the
> >> > > > > account
> >> > > > > > > ID
> >> > > > > > > > in
> >> > > > > > > > > > > > roleArn
> >> > > > > > > > > > > > > > or userArn would give those fields a different
> >> > meaning,
> >> > > > > so
> >> > > > > > > I’ve
> >> > > > > > > > > > left
> >> > > > > > > > > > > > them
> >> > > > > > > > > > > > > > absent for CLOUDFLARE_R2.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > I pursued Option B. The S3 configuration gains
> >> one
> >> > new
> >> > > > > > field,
> >> > > > > > > > > > > > > > credentialIssuer, with values STS
> >> > > > > > > > > > > > > > (default) and CLOUDFLARE_R2 and a realm
> >> allowlist
> >> > > > > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_ISSUERS that defaults
> >> > > > > > > > > > > > > > to [STS]. I left stsUnavailable untouched
> >> > (Prithvi) so
> >> > > > it
> >> > > > > > > still
> >> > > > > > > > > > means
> >> > > > > > > > > > > > > > "vend nothing". The
> >> > > > > > > > > > > > > > credentialIssuer is the explicit sign. The R2
> >> > config
> >> > > > > > rejects
> >> > > > > > > > > > > > > > stsUnavailable along with a host of other STS
> >> and
> >> > KMS
> >> > > > > > fields.
> >> > > > > > > > I'm
> >> > > > > > > > > > > > running
> >> > > > > > > > > > > > > > this in our non-production environment at the
> >> > moment.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Draft PR for stage 1 (config field, API,
> >> allowlist,
> >> > > > > gates):
> >> > > > > > > > > > > > > > https://github.com/apache/polaris/pull/5513.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > I still see the appeal of a separate R2
> >> > configuration,
> >> > > > as
> >> > > > > > > > Dmitri
> >> > > > > > > > > > > > > > suggested. My original reason for separating
> it
> >> > was to
> >> > > > > keep
> >> > > > > > > the
> >> > > > > > > > > > > > > R2-specific
> >> > > > > > > > > > > > > > configuration and validation together. Having
> >> > > > > > > > > > getCloudflareR2Endpoint
> >> > > > > > > > > > > > on
> >> > > > > > > > > > > > > > AwsStorageConfigurationInfo is one part I’d
> >> still
> >> > like
> >> > > > > > > feedback
> >> > > > > > > > > on.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > I wanted to put a working version of Option B
> in
> >> > front
> >> > > > of
> >> > > > > > > > > everyone
> >> > > > > > > > > > so
> >> > > > > > > > > > > > we
> >> > > > > > > > > > > > > > could discuss the actual changes. In my
> earlier
> >> > > > > > > implementation,
> >> > > > > > > > > the
> >> > > > > > > > > > > > > > separate config also meant a separate storage
> >> type
> >> > > > > sharing
> >> > > > > > > > > > S3FileIO.
> >> > > > > > > > > > > > That
> >> > > > > > > > > > > > > > was the tradeoff that led me to try Option B.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Do the diffs change anyone’s lean here? And
> are
> >> the
> >> > > > names
> >> > > > > > > > > > > > > credentialIssuer
> >> > > > > > > > > > > > > > / STS / CLOUDFLARE_R2 /
> >> > SUPPORTED_S3_CREDENTIAL_ISSUERS
> >> > > > > > > > > acceptable?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > > Austen
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Get Outlook for Mac<
> >> > https://aka.ms/GetOutlookForMac>
> >> > > > > > > > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> >> > > > > > > > > > > > > > Date: Sunday, September 13, 2026 at 19:50
> >> > > > > > > > > > > > > > To: dev@polaris.apache.org <
> >> dev@polaris.apache.org
> >> > >
> >> > > > > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support
> >> with
> >> > > > scoped
> >> > > > > > > > > credential
> >> > > > > > > > > > > > > vending
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Building on top of Ayush's and JB's
> suggestion,
> >> I
> >> > agree
> >> > > > > > that
> >> > > > > > > > > > > > jurisdiction
> >> > > > > > > > > > > > > > can live inside the existing endpoint config
> >> field.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > That leaves only Account ID, so potentially we
> >> > could
> >> > > > > store
> >> > > > > > it
> >> > > > > > > > in
> >> > > > > > > > > > user
> >> > > > > > > > > > > > or
> >> > > > > > > > > > > > > > role ARN. Note that userArn is currently
> unused
> >> in
> >> > OSS
> >> > > > > > code.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > That should allow us to keep existing
> >> > > > > > > > AwsStorageConfigurationInfo
> >> > > > > > > > > > for
> >> > > > > > > > > > > > all
> >> > > > > > > > > > > > > > S3 storage backend, but will require special
> >> > parsing of
> >> > > > > > > > user/role
> >> > > > > > > > > > ARN
> >> > > > > > > > > > > > > field
> >> > > > > > > > > > > > > > to distinguish R2 from the other systems.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > I'm still leaning towards the separate
> >> > > > > > > > > R2StorageConfigurationInfo,
> >> > > > > > > > > > > but
> >> > > > > > > > > > > > > I'd
> >> > > > > > > > > > > > > > be ok with overloading user/role ARN too if
> >> other
> >> > > > people
> >> > > > > > > prefer
> >> > > > > > > > > > > that. I
> >> > > > > > > > > > > > > > guess a key question in this context is
> whether
> >> R2
> >> > has
> >> > > > > the
> >> > > > > > > > > concept
> >> > > > > > > > > > of
> >> > > > > > > > > > > > > role
> >> > > > > > > > > > > > > > ARN at all.... but TBH I have no idea about
> >> that. I
> >> > > > hope
> >> > > > > > > Austen
> >> > > > > > > > > > might
> >> > > > > > > > > > > > be
> >> > > > > > > > > > > > > > able to clarify that.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Cheers,
> >> > > > > > > > > > > > > > Dmitri.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > On Sun, Sep 13, 2026 at 9:18 AM Prithvi S <
> >> > > > > > > > > > > prithvisivasankar@gmail.com
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Hi Austen,
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Welcome, and thanks for starting with a
> >> > [DISCUSS].
> >> > > > Best
> >> > > > > > way
> >> > > > > > > > to
> >> > > > > > > > > > > make a
> >> > > > > > > > > > > > > > first
> >> > > > > > > > > > > > > > > contribution :)
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > I think the thread is talking about two
> >> layers,
> >> > and
> >> > > > we
> >> > > > > > can
> >> > > > > > > > keep
> >> > > > > > > > > > > both
> >> > > > > > > > > > > > > > > answers.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > On the client side this should stay S3:
> s3://
> >> > > > > locations,
> >> > > > > > > s3.*
> >> > > > > > > > > > > > > properties,
> >> > > > > > > > > > > > > > > S3FileIO. Your PyIceberg / DuckDB / Iceberg
> >> Java
> >> > > > > results
> >> > > > > > > > > already
> >> > > > > > > > > > > show
> >> > > > > > > > > > > > > > that
> >> > > > > > > > > > > > > > > unmodified clients work, so a dedicated
> >> R2FileIO
> >> > does
> >> > > > > not
> >> > > > > > > > seem
> >> > > > > > > > > > > worth
> >> > > > > > > > > > > > > it.
> >> > > > > > > > > > > > > > > On the server side I lean toward Dmitri: a
> >> > distinct
> >> > > > > > > > > > > > > > > R2StorageConfigurationInfo and storage
> >> > integration.
> >> > > > > > > Issuance
> >> > > > > > > > is
> >> > > > > > > > > > not
> >> > > > > > > > > > > > > STS,
> >> > > > > > > > > > > > > > > and folding Cloudflare fields into
> >> > > > > > > > AwsStorageConfigurationInfo
> >> > > > > > > > > > > mixes
> >> > > > > > > > > > > > > > > unrelated concepts. The middle ground to
> avoid
> >> > is a
> >> > > > new
> >> > > > > > > type
> >> > > > > > > > > that
> >> > > > > > > > > > > is
> >> > > > > > > > > > > > > > still
> >> > > > > > > > > > > > > > > AWS-shaped underneath.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > One concrete caution if the Option B
> prototype
> >> > stays
> >> > > > on
> >> > > > > > the
> >> > > > > > > > S3
> >> > > > > > > > > > > > config:
> >> > > > > > > > > > > > > > > please do not overload stsUnavailable. Today
> >> that
> >> > > > flag
> >> > > > > > > skips
> >> > > > > > > > > > > > AssumeRole
> >> > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > disables vending. R2 wants the opposite.
> vend,
> >> > just
> >> > > > not
> >> > > > > > via
> >> > > > > > > > > STS.
> >> > > > > > > > > > so
> >> > > > > > > > > > > > > that
> >> > > > > > > > > > > > > > > flag would change meaning for existing
> MinIO /
> >> > NetApp
> >> > > > > > > > catalogs.
> >> > > > > > > > > > If
> >> > > > > > > > > > > we
> >> > > > > > > > > > > > > > stay
> >> > > > > > > > > > > > > > > on S3 config, we need an explicit vending
> >> > signal; a
> >> > > > > > > dedicated
> >> > > > > > > > > R2
> >> > > > > > > > > > > type
> >> > > > > > > > > > > > > > > avoids that trap.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > On config surface, jurisdiction can live in
> >> > endpoint
> >> > > > > > > > > (<account>.
> >> > > > > > > > > > > > > > > r2.cloudflarestorage.com vs
> >> <account>.eu.r2...).
> >> > > > > Account
> >> > > > > > > ID
> >> > > > > > > > is
> >> > > > > > > > > > > still
> >> > > > > > > > > > > > > > > needed
> >> > > > > > > > > > > > > > > for the JWT sub, so keeping an explicit
> >> > accountId and
> >> > > > > > > > deriving
> >> > > > > > > > > > > > audience
> >> > > > > > > > > > > > > > > from the endpoint seems enough. Parent
> tokens
> >> > should
> >> > > > > stay
> >> > > > > > > in
> >> > > > > > > > > > server
> >> > > > > > > > > > > > > > config
> >> > > > > > > > > > > > > > > via storageName.
> >> > > > > > > > > > > > > > > Local signing fits the vending model.
> >> > Fail-closed on
> >> > > > > > > > > cross-bucket
> >> > > > > > > > > > > and
> >> > > > > > > > > > > > > > mixed
> >> > > > > > > > > > > > > > > read/write grants is the right call. Prefix
> >> > claims
> >> > > > > should
> >> > > > > > > > > follow
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > > same
> >> > > > > > > > > > > > > > > path comparison Polaris already uses, so we
> do
> >> > not
> >> > > > > mint a
> >> > > > > > > > > > > credential
> >> > > > > > > > > > > > > > wider
> >> > > > > > > > > > > > > > > than the grant.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > +1 to small PRs: FileIO discriminator
> (#4486)
> >> > > > > separately
> >> > > > > > if
> >> > > > > > > > R2
> >> > > > > > > > > > > needs
> >> > > > > > > > > > > > > it,
> >> > > > > > > > > > > > > > > then management API + Java config, then
> >> vending,
> >> > > > Python
> >> > > > > > CLI
> >> > > > > > > > on
> >> > > > > > > > > > its
> >> > > > > > > > > > > > own.
> >> > > > > > > > > > > > > > > Sketching MinIO / Backblaze against the same
> >> > fields
> >> > > > is
> >> > > > > a
> >> > > > > > > > useful
> >> > > > > > > > > > > > design
> >> > > > > > > > > > > > > > > check; a generic non-STS SPI can wait for a
> >> > second
> >> > > > > > backend.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Regards,
> >> > > > > > > > > > > > > > > Prithvi S
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > On Fri, Sep 11, 2026 at 4:31 AM Dmitri
> >> > Bourlatchkov <
> >> > > > > > > > > > > > dimas@apache.org>
> >> > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Hi Austen,
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Thanks for starting this contribution!
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > After briefly looking through your PR [1]
> I
> >> > think
> >> > > > > > > > > > > > > > > > adding R2StorageConfigInfo makes sense.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Even though "aws" in
> >> > AwsStorageConfigurationInfo is
> >> > > > > > > longer
> >> > > > > > > > > > > implies
> >> > > > > > > > > > > > > AWS
> >> > > > > > > > > > > > > > > > services, and MinIO, Rust, Ozone are
> >> supported
> >> > just
> >> > > > > as
> >> > > > > > > > well,
> >> > > > > > > > > R2
> >> > > > > > > > > > > > > appears
> >> > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > be significantly different from STS-based
> S3
> >> > > > systems
> >> > > > > > > that a
> >> > > > > > > > > > > > separate
> >> > > > > > > > > > > > > > > config
> >> > > > > > > > > > > > > > > > is probably the most natural way to
> support
> >> it
> >> > in
> >> > > > the
> >> > > > > > > > Polaris
> >> > > > > > > > > > > > > codebase.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > The alternative, I guess, is to add R2
> >> > accountId
> >> > > > and
> >> > > > > > > > > > jurisdiction
> >> > > > > > > > > > > > as
> >> > > > > > > > > > > > > > > > optional properties to
> >> > AwsStorageConfigurationInfo
> >> > > > > (or
> >> > > > > > > > > overload
> >> > > > > > > > > > > > > > existing
> >> > > > > > > > > > > > > > > > properties), but that looks like mixing
> >> > unrelated
> >> > > > > > > concepts
> >> > > > > > > > > > > > together.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Adding a generic property bag to
> >> > > > > > > > AwsStorageConfigurationInfo
> >> > > > > > > > > is
> >> > > > > > > > > > > not
> >> > > > > > > > > > > > > > > > convenient because
> >> > > > > > PolarisStorageIntegrationProviderImpl
> >> > > > > > > > will
> >> > > > > > > > > > > > > probably
> >> > > > > > > > > > > > > > > have
> >> > > > > > > > > > > > > > > > to know how to interpret it in order to
> >> > redirect
> >> > > > > > > credential
> >> > > > > > > > > > > vending
> >> > > > > > > > > > > > > > > > requests to R2-specific code.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > I wonder whether
> R2StorageConfigurationInfo
> >> and
> >> > > > > > > > > > > > > > > AwsStorageConfigurationInfo
> >> > > > > > > > > > > > > > > > (and corresponding "storage integration"
> >> > classes)
> >> > > > > could
> >> > > > > > > > > share a
> >> > > > > > > > > > > > > common
> >> > > > > > > > > > > > > > > base
> >> > > > > > > > > > > > > > > > class for common properties (e.g.
> endpoint).
> >> > Even
> >> > > > > > though
> >> > > > > > > > > > > "endpoint"
> >> > > > > > > > > > > > > > might
> >> > > > > > > > > > > > > > > > not be relevant to R2 in the cloud, the
> fact
> >> > that
> >> > > > > > > S3FileIO
> >> > > > > > > > > uses
> >> > > > > > > > > > > it,
> >> > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > R2
> >> > > > > > > > > > > > > > > > is accessed via S3FileIO means we should
> >> > probably
> >> > > > > allow
> >> > > > > > > > users
> >> > > > > > > > > > to
> >> > > > > > > > > > > > > > > configure
> >> > > > > > > > > > > > > > > > the full set of FileIO properties just in
> >> case.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > I think it is fine for different
> >> > > > > > > PolarisStorageIntegration
> >> > > > > > > > > > > > > > > implementations
> >> > > > > > > > > > > > > > > > to produce the same kind of
> >> > StorageAccessConfig (S3
> >> > > > > in
> >> > > > > > > this
> >> > > > > > > > > > case)
> >> > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > thus
> >> > > > > > > > > > > > > > > > cause S3FileIO to be used by clients
> >> (including
> >> > > > > > Polaris'
> >> > > > > > > > own
> >> > > > > > > > > > > > storage
> >> > > > > > > > > > > > > > > access
> >> > > > > > > > > > > > > > > > paths).
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Changes in StorageTypeFileIO in [1] look a
> >> bit
> >> > > > > > concerning
> >> > > > > > > > to
> >> > > > > > > > > > me.
> >> > > > > > > > > > > I
> >> > > > > > > > > > > > > > wonder
> >> > > > > > > > > > > > > > > > if this code could be refactored to avoid
> >> > > > > dependencies
> >> > > > > > on
> >> > > > > > > > > > storage
> >> > > > > > > > > > > > > > config
> >> > > > > > > > > > > > > > > > types completely.... but TBH, I did not
> look
> >> > too
> >> > > > > deeply
> >> > > > > > > > into
> >> > > > > > > > > > this
> >> > > > > > > > > > > > > > today.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Anticipating future PRs for this in the
> >> Polaris
> >> > > > repo,
> >> > > > > > I'd
> >> > > > > > > > > like
> >> > > > > > > > > > to
> >> > > > > > > > > > > > ask
> >> > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > separate Python code changes from java
> >> changes
> >> > > > since
> >> > > > > > > these
> >> > > > > > > > > > areas
> >> > > > > > > > > > > > > > usually
> >> > > > > > > > > > > > > > > > attract different reviewers :)
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > From my POV, management API and matching
> >> java
> >> > > > changes
> >> > > > > > can
> >> > > > > > > > be
> >> > > > > > > > > in
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > > > same
> >> > > > > > > > > > > > > > > > PR. I'd say docs should come later (to
> keep
> >> PRs
> >> > > > > small).
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > All in all, thank you again and I hope
> this
> >> > > > > > contribution
> >> > > > > > > > > lands
> >> > > > > > > > > > in
> >> > > > > > > > > > > > > > Polaris
> >> > > > > > > > > > > > > > > > :)
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > [1]
> >> > https://github.com/deepdishgary/polaris/pull/1
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Cheers,
> >> > > > > > > > > > > > > > > > Dmitri.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > On Thu, Sep 10, 2026 at 1:52 PM
> >> Jean-Baptiste
> >> > > > Onofré
> >> > > > > <
> >> > > > > > > > > > > > > jb@nanthrax.net>
> >> > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Hi Austen,
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Nice discussion! Thanks for that, and
> >> welcome
> >> > > > > aboard
> >> > > > > > :)
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > I think it makes sense to have R2 built
> >> > under the
> >> > > > > > > > existing
> >> > > > > > > > > S3
> >> > > > > > > > > > > > > storage
> >> > > > > > > > > > > > > > > > > configuration. It matches how Polaris is
> >> > already
> >> > > > > > > > > structured:
> >> > > > > > > > > > > > > > > > > StorageType is keyed on the URI scheme
> >> > (s3://,
> >> > > > > > s3a://),
> >> > > > > > > > and
> >> > > > > > > > > > > > > > > > > AwsStorageConfigurationInfo already
> >> carries
> >> > > > > endpoint,
> >> > > > > > > > > > > > stsEndpoint,
> >> > > > > > > > > > > > > > > > > pathStyleAccess and, importantly, an
> >> > > > stsUnavailable
> >> > > > > > > flag.
> >> > > > > > > > > > > > > > > > > So, what you need is largely there
> >> already:
> >> > > > correct
> >> > > > > > me
> >> > > > > > > if
> >> > > > > > > > > I'm
> >> > > > > > > > > > > > > wrong,
> >> > > > > > > > > > > > > > > > > R2 is essentially "S3 compatible store
> >> where
> >> > STS
> >> > > > is
> >> > > > > > > > > > > unavailable,
> >> > > > > > > > > > > > > plus
> >> > > > > > > > > > > > > > > > > a way to actually vend credentials in
> that
> >> > case".
> >> > > > > > Going
> >> > > > > > > > > this
> >> > > > > > > > > > > > route
> >> > > > > > > > > > > > > > > > > also keeps this work "isolated" from the
> >> > > > > > > > > > > FileIO-per-storage-type
> >> > > > > > > > > > > > > > > > > discussion.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > The interesting design question is a
> >> > pluggable
> >> > > > > > > credential
> >> > > > > > > > > > > vending
> >> > > > > > > > > > > > > > path
> >> > > > > > > > > > > > > > > > > for S3 compatible stores when STS is not
> >> > > > available,
> >> > > > > > > with
> >> > > > > > > > > > > > > Cloudflare's
> >> > > > > > > > > > > > > > > > > local signing as the first concrete
> >> > > > implementation,
> >> > > > > > > > rather
> >> > > > > > > > > > than
> >> > > > > > > > > > > > R2
> >> > > > > > > > > > > > > > > > > specific code. It would be good to
> sketch
> >> how
> >> > > > MinIO
> >> > > > > > > would
> >> > > > > > > > > > slot
> >> > > > > > > > > > > > into
> >> > > > > > > > > > > > > > > > > the same config fields before the
> vending
> >> > path
> >> > > > > > hardens
> >> > > > > > > > > around
> >> > > > > > > > > > > R2.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > I believe we have to check if Cloudflare
> >> > account
> >> > > > ID
> >> > > > > > and
> >> > > > > > > > > > > > > jurisdiction
> >> > > > > > > > > > > > > > > > > really need dedicated config fields, or
> >> can
> >> > we
> >> > > > use
> >> > > > > > the
> >> > > > > > > > > > endpoint
> >> > > > > > > > > > > > > value
> >> > > > > > > > > > > > > > > > > (with documentation)? Keeping the
> >> > configuration
> >> > > > > > surface
> >> > > > > > > > > > minimal
> >> > > > > > > > > > > > > will
> >> > > > > > > > > > > > > > > > > help the design generalize.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > I would love to discuss two things:
> >> > > > > > > > > > > > > > > > > 1. Polaris shipping a Cloudflare API
> token
> >> > and
> >> > > > > > locally
> >> > > > > > > > > > minting
> >> > > > > > > > > > > > > scoped
> >> > > > > > > > > > > > > > > > > credentials makes it effectively an STS
> >> for
> >> > R2,
> >> > > > > which
> >> > > > > > > > > > increases
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > blast radius of a server compromise.
> Your
> >> > "reject
> >> > > > > > > rather
> >> > > > > > > > > than
> >> > > > > > > > > > > > > > boarden"
> >> > > > > > > > > > > > > > > > > handling of unrepresentable scopes is
> the
> >> > right
> >> > > > > call.
> >> > > > > > > > > > > > > > > > > 2. The PyIceberg/DuckDB/Java results are
> >> > > > promising.
> >> > > > > > > Just
> >> > > > > > > > > > > curious
> >> > > > > > > > > > > > > > about
> >> > > > > > > > > > > > > > > > > Spark or Trino.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > We love small, well-scoped PRs. I would
> >> > suggest
> >> > > > to
> >> > > > > > > start
> >> > > > > > > > > with
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > > S3
> >> > > > > > > > > > > > > > > > > config and the management API as first
> >> > shoot, and
> >> > > > > we
> >> > > > > > > can
> >> > > > > > > > > > follow
> >> > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > the credential vending implementation.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Regards
> >> > > > > > > > > > > > > > > > > JB
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > On Wed, Sep 9, 2026 at 8:18 PM Austen
> >> Tomek
> >> > > > > > > > > > > > > > > > > <Au...@chicagotrading.com>
> wrote:
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Hi all,
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > I’m Austen Tomek from Chicago Trading
> >> > Company.
> >> > > > We
> >> > > > > > use
> >> > > > > > > > > > Polaris
> >> > > > > > > > > > > > for
> >> > > > > > > > > > > > > > our
> >> > > > > > > > > > > > > > > > > Iceberg catalogs and we're evaluating
> >> > Cloudflare
> >> > > > R2
> >> > > > > > as
> >> > > > > > > an
> >> > > > > > > > > > > object
> >> > > > > > > > > > > > > > store
> >> > > > > > > > > > > > > > > > (the
> >> > > > > > > > > > > > > > > > > no egress fees make it a very compelling
> >> > > > product).
> >> > > > > > I’d
> >> > > > > > > > like
> >> > > > > > > > > > to
> >> > > > > > > > > > > > > > discuss
> >> > > > > > > > > > > > > > > > > contributing R2 support and credential
> >> > vending to
> >> > > > > > > Polaris
> >> > > > > > > > > > with
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > > goal
> >> > > > > > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > getting feedback before opening a
> >> > > > ready-for-review
> >> > > > > > > > upstream
> >> > > > > > > > > > PR.
> >> > > > > > > > > > > > > This
> >> > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > my
> >> > > > > > > > > > > > > > > > > first contribution, so trying to do this
> >> > right.
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > R2 exposes S3-compatible APIs but it
> >> does
> >> > not
> >> > > > > > provide
> >> > > > > > > > AWS
> >> > > > > > > > > > > STS.
> >> > > > > > > > > > > > > > > > > Cloudflare instead supports these
> >> > short-lived,
> >> > > > > scoped
> >> > > > > > > > > > > credentials
> >> > > > > > > > > > > > > > > > generated
> >> > > > > > > > > > > > > > > > > by locally signed JWT with the server
> >> > holding the
> >> > > > > > > parent
> >> > > > > > > > > API
> >> > > > > > > > > > > > token
> >> > > > > > > > > > > > > > [1].
> >> > > > > > > > > > > > > > > > > This allows us to retain Polaris’s
> >> > authorization
> >> > > > > and
> >> > > > > > > > > > > > > > credential-vending
> >> > > > > > > > > > > > > > > > > model while storing our table data in
> R2.
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > I have a prototype with the following
> >> > design:
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >   *
> >> > > > > > > > > > > > > > > > > > An opt-in R2 storage type, with
> catalog
> >> > > > > > configuration
> >> > > > > > > > > > > > identifying
> >> > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > Cloudflare account and optional
> >> jurisdiction.
> >> > > > > Parent
> >> > > > > > > > > > > credentials
> >> > > > > > > > > > > > > > remain
> >> > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > server configuration and can be selected
> >> > through
> >> > > > > > > > > storageName.
> >> > > > > > > > > > > > > > > > > >   *
> >> > > > > > > > > > > > > > > > > > Local credential generation using
> >> > Cloudflare’s
> >> > > > > > > > documented
> >> > > > > > > > > > > > format.
> >> > > > > > > > > > > > > > > > > Credentials expire, are restricted to
> one
> >> > bucket,
> >> > > > > and
> >> > > > > > > > carry
> >> > > > > > > > > > > > prefix
> >> > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > access scopes derived from Polaris’s
> >> location
> >> > > > > grants.
> >> > > > > > > The
> >> > > > > > > > > > > > > integration
> >> > > > > > > > > > > > > > > > uses
> >> > > > > > > > > > > > > > > > > Polaris’s existing credential cache.
> >> > > > > > > > > > > > > > > > > >   *
> >> > > > > > > > > > > > > > > > > > Credentials returned through the
> >> existing
> >> > > > Iceberg
> >> > > > > > > REST
> >> > > > > > > > > > > vending
> >> > > > > > > > > > > > > flow
> >> > > > > > > > > > > > > > > as
> >> > > > > > > > > > > > > > > > > s3.* properties, including the endpoint,
> >> > session
> >> > > > > > token,
> >> > > > > > > > and
> >> > > > > > > > > > > > expiry.
> >> > > > > > > > > > > > > > > > > Compatible clients use their existing S3
> >> > FileIO.
> >> > > > > > > > > > > > > > > > > >   *
> >> > > > > > > > > > > > > > > > > > Conservative handling of scopes the
> >> > > > > implementation
> >> > > > > > > > cannot
> >> > > > > > > > > > > > > > represent:
> >> > > > > > > > > > > > > > > > > cross-bucket and mixed read/write grants
> >> are
> >> > > > > rejected
> >> > > > > > > > > rather
> >> > > > > > > > > > > than
> >> > > > > > > > > > > > > > > > > broadened. R2 is excluded from the
> default
> >> > > > > > > > > > > > supported-storage-types
> >> > > > > > > > > > > > > > > list.
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > The main design questions I feel this
> >> runs
> >> > > > into:
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >   *
> >> > > > > > > > > > > > > > > > > > Should this be a distinct storage type
> >> or a
> >> > > > > > > > > > > credential-vending
> >> > > > > > > > > > > > > > > > mechanism
> >> > > > > > > > > > > > > > > > > selected through the existing S3
> >> > configuration?
> >> > > > > > > > > > > > > > > > > >      *
> >> > > > > > > > > > > > > > > > > > I chose a separate type because the
> >> > identity,
> >> > > > > > > > credential
> >> > > > > > > > > > > > > > generation,
> >> > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > scoping model differ from AWS STS, as
> this
> >> > seemed
> >> > > > > > more
> >> > > > > > > > > > > > appropriate
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > There is a draft preview PR against my
> >> fork
> >> > > > [2],
> >> > > > > > > > > including
> >> > > > > > > > > > > > > > > > > implementation, management API changes,
> >> CLI
> >> > > > > support,
> >> > > > > > > > tests,
> >> > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > documentation. It also discloses the AI
> >> > > > assistance
> >> > > > > > used
> >> > > > > > > > > > during
> >> > > > > > > > > > > > > > > > > implementation. I leaned heavily on
> Fable
> >> > 5.1 for
> >> > > > > > this
> >> > > > > > > > > > > > > implementation
> >> > > > > > > > > > > > > > > so
> >> > > > > > > > > > > > > > > > I
> >> > > > > > > > > > > > > > > > > want to be upfront about it.
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > We are running the implementation in a
> >> > > > > > non-production
> >> > > > > > > > > > > > environment
> >> > > > > > > > > > > > > > > right
> >> > > > > > > > > > > > > > > > > now as a POC. Validation includes live
> R2
> >> > > > > > reads/writes,
> >> > > > > > > > > > > multipart
> >> > > > > > > > > > > > > > > > > operations, negative scope-boundary
> tests,
> >> > our
> >> > > > > > internal
> >> > > > > > > > > > Python
> >> > > > > > > > > > > > > client
> >> > > > > > > > > > > > > > > > > regression matrix, and credential
> >> > > > rotation/recovery
> >> > > > > > > > > > exercises.
> >> > > > > > > > > > > > The
> >> > > > > > > > > > > > > > > > upstream
> >> > > > > > > > > > > > > > > > > CI workflow has passed on the fork.
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > The change also encounters the
> existing
> >> > > > > assumption
> >> > > > > > > > that a
> >> > > > > > > > > > > > FileIO
> >> > > > > > > > > > > > > > > > > implementation identifies exactly one
> >> storage
> >> > > > type:
> >> > > > > > > both
> >> > > > > > > > S3
> >> > > > > > > > > > and
> >> > > > > > > > > > > > R2
> >> > > > > > > > > > > > > > use
> >> > > > > > > > > > > > > > > > > S3FileIO. This overlaps with #4486 [3].
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > I’d particularly appreciate feedback
> on:
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >   1.
> >> > > > > > > > > > > > > > > > > > Should R2 have its own storage
> >> > > > > configuration/type,
> >> > > > > > or
> >> > > > > > > > > > should
> >> > > > > > > > > > > > > > non-STS
> >> > > > > > > > > > > > > > > > > vending be modeled within the existing
> S3
> >> > > > > > integration?
> >> > > > > > > > > > > > > > > > > >   2.
> >> > > > > > > > > > > > > > > > > > Does the local-signing approach fit
> >> > Polaris’s
> >> > > > > > > > > > > > credential-vending
> >> > > > > > > > > > > > > > > model?
> >> > > > > > > > > > > > > > > > > Any additional constraints or
> validations
> >> > > > > necessary?
> >> > > > > > > > > > > > > > > > > >   3.
> >> > > > > > > > > > > > > > > > > > Would you prefer the shared FileIO
> >> > validation
> >> > > > > > change
> >> > > > > > > to
> >> > > > > > > > > > land
> >> > > > > > > > > > > > > > > > separately,
> >> > > > > > > > > > > > > > > > > and how should we sequence the
> management
> >> > API and
> >> > > > > R2
> >> > > > > > > > > > > integration
> >> > > > > > > > > > > > > > > changes?
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >   Happy to adapt the design and split
> >> the
> >> > > > > > > contribution
> >> > > > > > > > > into
> >> > > > > > > > > > > > more
> >> > > > > > > > > > > > > > > > focused
> >> > > > > > > > > > > > > > > > > PRs. Wanted to start the discussion
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >   Thanks,
> >> > > > > > > > > > > > > > > > > >   Austen
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >   [1]
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > >
> >> https://developers.cloudflare.com/r2/api/s3/temporary-credentials/
> >> > > > > > > > > > > > > > > > > >   [2]
> >> > > > > > https://github.com/deepdishgary/polaris/pull/1
> >> > > > > > > > > > > > > > > > > >   [3]
> >> > > > > > https://github.com/apache/polaris/issues/4486
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Get Outlook for Mac<
> >> > > > > > https://aka.ms/GetOutlookForMac>
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > This electronic mail message and any
> >> > attached
> >> > > > > files
> >> > > > > > > > > contain
> >> > > > > > > > > > > > > > > information
> >> > > > > > > > > > > > > > > > > intended for the exclusive use of the
> >> > individual
> >> > > > or
> >> > > > > > > > entity
> >> > > > > > > > > to
> >> > > > > > > > > > > > whom
> >> > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > > addressed and may contain information
> >> that is
> >> > > > > > > > proprietary,
> >> > > > > > > > > > > > > > confidential
> >> > > > > > > > > > > > > > > > > and/or exempt from disclosure under
> >> > applicable
> >> > > > law.
> >> > > > > > If
> >> > > > > > > > you
> >> > > > > > > > > > are
> >> > > > > > > > > > > > not
> >> > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > intended recipient, you are hereby
> >> notified
> >> > that
> >> > > > > any
> >> > > > > > > > > viewing,
> >> > > > > > > > > > > > > > copying,
> >> > > > > > > > > > > > > > > > > disclosure or distribution of this
> >> > information
> >> > > > may
> >> > > > > be
> >> > > > > > > > > subject
> >> > > > > > > > > > > to
> >> > > > > > > > > > > > > > legal
> >> > > > > > > > > > > > > > > > > restriction or sanction. Please notify
> the
> >> > > > sender,
> >> > > > > by
> >> > > > > > > > > > > electronic
> >> > > > > > > > > > > > > mail
> >> > > > > > > > > > > > > > > or
> >> > > > > > > > > > > > > > > > > telephone, of any unintended recipients
> >> and
> >> > > > delete
> >> > > > > > the
> >> > > > > > > > > > original
> >> > > > > > > > > > > > > > message
> >> > > > > > > > > > > > > > > > > without making any copies.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > This electronic mail message and any attached
> >> files
> >> > > > > contain
> >> > > > > > > > > > > information
> >> > > > > > > > > > > > > > intended for the exclusive use of the
> >> individual or
> >> > > > > entity
> >> > > > > > to
> >> > > > > > > > > whom
> >> > > > > > > > > > it
> >> > > > > > > > > > > > is
> >> > > > > > > > > > > > > > addressed and may contain information that is
> >> > > > > proprietary,
> >> > > > > > > > > > > confidential
> >> > > > > > > > > > > > > > and/or exempt from disclosure under applicable
> >> > law. If
> >> > > > > you
> >> > > > > > > are
> >> > > > > > > > > not
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > > > intended recipient, you are hereby notified
> that
> >> > any
> >> > > > > > viewing,
> >> > > > > > > > > > > copying,
> >> > > > > > > > > > > > > > disclosure or distribution of this information
> >> may
> >> > be
> >> > > > > > subject
> >> > > > > > > > to
> >> > > > > > > > > > > legal
> >> > > > > > > > > > > > > > restriction or sanction. Please notify the
> >> sender,
> >> > by
> >> > > > > > > > electronic
> >> > > > > > > > > > mail
> >> > > > > > > > > > > > or
> >> > > > > > > > > > > > > > telephone, of any unintended recipients and
> >> delete
> >> > the
> >> > > > > > > original
> >> > > > > > > > > > > message
> >> > > > > > > > > > > > > > without making any copies.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > This electronic mail message and any attached files
> >> > contain
> >> > > > > > > > information
> >> > > > > > > > > > > intended for the exclusive use of the individual or
> >> > entity to
> >> > > > > > whom
> >> > > > > > > it
> >> > > > > > > > > is
> >> > > > > > > > > > > addressed and may contain information that is
> >> > proprietary,
> >> > > > > > > > confidential
> >> > > > > > > > > > > and/or exempt from disclosure under applicable law.
> If
> >> > you
> >> > > > are
> >> > > > > > not
> >> > > > > > > > the
> >> > > > > > > > > > > intended recipient, you are hereby notified that any
> >> > viewing,
> >> > > > > > > > copying,
> >> > > > > > > > > > > disclosure or distribution of this information may
> be
> >> > subject
> >> > > > > to
> >> > > > > > > > legal
> >> > > > > > > > > > > restriction or sanction. Please notify the sender,
> by
> >> > > > > electronic
> >> > > > > > > mail
> >> > > > > > > > > or
> >> > > > > > > > > > > telephone, of any unintended recipients and delete
> the
> >> > > > original
> >> > > > > > > > message
> >> > > > > > > > > > > without making any copies.
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > This electronic mail message and any attached files
> >> contain
> >> > > > > > > information
> >> > > > > > > > > > intended for the exclusive use of the individual or
> >> entity
> >> > to
> >> > > > > whom
> >> > > > > > it
> >> > > > > > > > is
> >> > > > > > > > > > addressed and may contain information that is
> >> proprietary,
> >> > > > > > > confidential
> >> > > > > > > > > > and/or exempt from disclosure under applicable law. If
> >> you
> >> > are
> >> > > > > not
> >> > > > > > > the
> >> > > > > > > > > > intended recipient, you are hereby notified that any
> >> > viewing,
> >> > > > > > > copying,
> >> > > > > > > > > > disclosure or distribution of this information may be
> >> > subject
> >> > > > to
> >> > > > > > > legal
> >> > > > > > > > > > restriction or sanction. Please notify the sender, by
> >> > > > electronic
> >> > > > > > mail
> >> > > > > > > > or
> >> > > > > > > > > > telephone, of any unintended recipients and delete the
> >> > original
> >> > > > > > > message
> >> > > > > > > > > > without making any copies.
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > This electronic mail message and any attached files
> contain
> >> > > > > information
> >> > > > > > > > intended for the exclusive use of the individual or entity
> >> to
> >> > whom
> >> > > > it
> >> > > > > > is
> >> > > > > > > > addressed and may contain information that is proprietary,
> >> > > > > confidential
> >> > > > > > > > and/or exempt from disclosure under applicable law. If you
> >> are
> >> > not
> >> > > > > the
> >> > > > > > > > intended recipient, you are hereby notified that any
> >> viewing,
> >> > > > > copying,
> >> > > > > > > > disclosure or distribution of this information may be
> >> subject
> >> > to
> >> > > > > legal
> >> > > > > > > > restriction or sanction. Please notify the sender, by
> >> > electronic
> >> > > > mail
> >> > > > > > or
> >> > > > > > > > telephone, of any unintended recipients and delete the
> >> original
> >> > > > > message
> >> > > > > > > > without making any copies.
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> >
> >>
> >
>