[DISCUSS] Multiple StorageConfigurationInfos per catalog

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi folks,

I would like to discuss a use case and potentially an enhancement on
Polaris.

The use case is as follows:
1. I have one Polaris catalog.
2. I have a S3 StorageConfigurationInfo (containing my S3 credentials,
allowed locations, region, endpoint).
3. Optionally, at namespace level, I can have base-location, inheriting the
catalog's storage credentials.
4. A table level, I can specify a location/base-location, again inheriting
the catalog's storage credentials.
It means that a table can have its data in s3://bucket-a/table1/, and
another table can use s3://bucket-b/table2/, as long as both buckets are
accessible with the same catalog-level credentials and are in the catalog's
allowed locations.

Now, I would like to have (again in a single catalog), one table on a S3,
another table on another S3.
The credential vending walks up the entity hierarchy
(FileIOUtil.findStorageInfoFromHierarchy) and uses the first
StorageConfigurationInfo it finds, which is always at the catalog level.

The only viable option today is to create separate catalogs, each with its
own S3 storage configuration pointing to a different bucket.

I would like to introduce an "identifier" on StorageConfigurationInfo and
having several StorageConfigurationInfos defined at catalog level.
Then, we could add an extra table property to define the
StorageConfigurationInfo identifier to us.

Thoughts ?

I would be happy to work on a proposal about that if there is interest.

Regards
JB

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Yufei Gu <fl...@gmail.com>]

Thanks a lot for working on it, Rishi! Going directly with phase 2
makes a lot of sense. Did one round of review on the PR. Let's iterate
on it.

> Storage mappings let an admin map the name of a namespace or table to a storage configuration, even before the object exists.

Mapping itself is good to have. But I'm not sure how an admin
pre-creates a mapping between a storage config and a table/ns when the
table/ns doesn't exist.
We cannot use the entity ID since it is allocated at creation time. We
also cannot use the table identifier either. Two reasons: 1. The
mapping is expressed as a pair of (Storage config entity id, table
entity id). Pls be aware that we need the table entity ID here so that
renaming won't break the mapping. 2. It isn't practical for an admin
to guess which table users will create.

Using a table property isn't perfect, but it is acceptable if we want
to avoid introducing a new DDL for storage configuration mapping.

On Sun, Nov 1, 2026 at 10:51 AM Srinivas Rishindra
<sr...@gmail.com> wrote:
>
> Hi Yufei, all,
>
> I've opened a PR with the API spec for storage configurations as a
> dedicated, catalog-scoped entity:
> https://github.com/apache/polaris/pull/5943. It is spec only and replaces
> #5721 and #5556.
>
> In short:
>
>   - Storage configurations are named entities under a catalog, each with
> its own settings and optional base location. They are immutable. The
> catalog's storageConfigInfo stays the default.
>   - Storage mappings let an admin map the name of a namespace or table to a
> storage configuration, even before the object exists. Any create under that
> name then uses it, including plain Iceberg REST creates and CTAS, with no
> client changes.
>   - A Polaris create API maps and creates in one call, using the Iceberg
> request and response bodies.
>   - Each storage configuration can reuse server-side credentials by name
> through the existing storageName field from #3409. Credentials stay in
> server configuration.
>   - Creating a storage configuration only runs static checks, because
> access is often granted afterwards. A separate validate call tests access
> to its locations on request.
>
> Yufei, mappings differ from the table property you suggested and from the
> pending bindings in my last email, so that's the part I'd most like
> feedback on. Which cloud identity a table gets is an admin's decision, and
> with a property anyone who can create a table would choose it. The PR
> spells out the rules for renames, stale mappings and cleanup.
>
> The storageName lookup exists only for AWS today. I'm also working on
> separate PRs to add it for GCS and Azure, and the spec assumes them.
>
> Please take a look.
>
> Thanks,
> Rishi
>
>
> On Mon, Oct 26, 2026 at 9:02 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > Thanks, Rishi! This looks good overall.
> >
> > I'm good with Option B. Catalog scoped storage configurations also make
> > sense to me. I agree that having a dedicated entity gives us a cleaner
> > model for authorization, versioning, and future extensions.
> >
> > Regarding the binding issue, I wonder if we can keep things simpler by
> > allowing a storage configuration to be specified during table creation(for
> > example, specified in a table property, so that engines like Spark don't
> > have to adopt a new Polaris-specific plugin), while continuing to inherit
> > the parent's configuration by default.
> >
> > The pending binding approach seems more complicated, especially around
> > renames, stale bindings, and cleanup. I'd prefer to avoid introducing that
> > complexity unless we have a strong use case that requires it.
> >
> > I'm good with moving forward with the design doc based on Option B.
> >
> >
> >
> > Yufei
> >
> >
> > On Sun, Oct 25, 2026 at 8:49 PM Srinivas Rishindra <sririshindra@gmail.com
> > >
> > wrote:
> >
> > > Hi Yufei,
> > >
> > > Thanks for writing this up. I went through your points and I agree: let's
> > > skip Phase 1 and make storage configurations their own entities.
> > >
> > > On your points:
> > >
> > >    1. Default base location. This goes beyond create-table. With default
> > >    settings a namespace can't have a custom location, so every namespace
> > >    location is derived from the catalog's default-base-location. A
> > > namespace
> > >    bound to an Azure configuration in an S3 catalog would get an s3://
> > >    location, and every table in it would fail the allowed-locations
> > check.
> > > The
> > >    only workaround is turning on ALLOW_NAMESPACE_CUSTOM_LOCATION and
> > > setting
> > >    each namespace's location by hand. An entity doesn't fix this by
> > itself
> > >    either. I'd give each configuration an optional base location, use it
> > > for
> > >    namespaces and tables bound before they're created (more below), and
> > > refuse
> > >    to bind an existing namespace or table whose location isn't inside the
> > >    configuration's allowed locations.
> > >
> > > 2, 3 and 5. Agreed. On 3: #5721 already moved to set/remove fields, but
> > > it's
> > > still one catalog entity with one version, so two admins editing
> > different
> > > configurations conflict.
> > >
> > >    4. Access control. Agreed, and this is the real cost of skipping Phase
> > >    1: we deferred AuthZ in June to unblock it, and now we need it up
> > front.
> > >    I'd follow the policy model: create, read, write, drop and list
> > > privileges
> > >    on the configuration, and attach as a pair (STORAGE_CONFIG_ATTACH on
> > the
> > >    configuration plus a dedicated attach privilege on the namespace or
> > > table),
> > >    all granted to catalog roles. It shouldn't be TABLE_WRITE_PROPERTIES,
> > >    because TABLE_WRITE_DATA includes it and every writer could move a
> > > table to
> > >    another identity. I'd leave out detach. Unlike a policy, unbinding a
> > > table
> > >    that has data only points it at a configuration that can't reach its
> > > files.
> > >    Changing a binding is a rebind, with the same location check as above.
> > >
> > > On your questions:
> > >
> > >    1.
> > >
> > >    Skipping Phase 1: +1.
> > >    2.
> > >
> > >    Option B. Policies already do most of what we need, but they fit
> > storage
> > >    poorly. A policy lives under a namespace, and POLICY_CREATE is a
> > > namespace
> > >    privilege. Its content is an untyped JSON string. Applicable policies
> > > are
> > >    returned to anyone with TABLE_READ_PROPERTIES, so role ARNs and
> > external
> > >    IDs would be visible to every table reader. And resolving them costs a
> > >    lookup per path level on every loadTable. I'd copy the patterns
> > instead
> > >    (attach checked on both sides, versioned updates) into a
> > STORAGE_CONFIG
> > >    entity, and keep the binding as a configuration name on the namespace
> > or
> > >    table, as you and Dmitri suggested in June.
> > >    3.
> > >
> > >    Scope. Catalog-scoped. A configuration holds an identity (role ARN and
> > >    external ID, Azure tenant, GCS service account) and a fence (allowed
> > >    locations, base location). The identity is the part worth sharing. The
> > >    fence has to stay per catalog, because the cross-catalog overlap check
> > > is
> > >    what stops one catalog's admin from getting credentials for another
> > >    catalog's files. A configuration shared across catalogs, locations
> > >    included, would fail that check, and the only way around it is
> > >    ALLOW_OVERLAPPING_CATALOG_URLS, which is realm-wide. So I'd start
> > >    catalog-scoped with the identity inline. Later the identity can
> > become a
> > >    realm-level object, managed by service_admin, that configurations
> > > reference
> > >    by name. That's additive, and close to how Unity Catalog separates
> > >    credentials from external locations.
> > >
> > > One more thing worth changing while we're at it: how a table ends up on a
> > > configuration other than its parent's. A binding needs an existing table,
> > > but
> > > Polaris writes the first metadata.json (and, for CTAS, vends the first
> > > credentials) with the parent's configuration. A Polaris-specific
> > > create-and-bind
> > > call would get around, but Spark, Trino and dbt can't use it directly.
> > >
> > > Instead (or in addition), an admin could bind a name ahead of time:
> > > "finance.ledger
> > > uses configuration finance". When a client then creates finance.ledger
> > > through the normal createTable, Polaris uses that configuration for the
> > > location, the first write and the credentials, and moves the binding onto
> > > the
> > > new table. Namespaces work the same way, which is what fixes point 1. The
> > > obvious concerns are stale bindings, renames, drop and re-create, and who
> > > owns a name that doesn't exist yet. I think each has an answer: a pending
> > > binding is used up at create and from then on behaves like any other
> > > binding, dropping the configuration deletes its pending bindings, and the
> > > namespace owns the names under it, so binding ahead needs the same
> > > privileges as binding the namespace. It does need a small table for
> > pending
> > > bindings. If that's too much, we could allow it for namespaces only;
> > tables
> > > then follow their namespace.
> > >
> > > If this sounds right, I'll update the design doc and share it here before
> > > opening a spec PR. I'll keep #5721 open until then.
> > >
> >

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Srinivas Rishindra <sr...@gmail.com>]

Hi Yufei, all,

I've opened a PR with the API spec for storage configurations as a
dedicated, catalog-scoped entity:
https://github.com/apache/polaris/pull/5943. It is spec only and replaces
#5721 and #5556.

In short:

  - Storage configurations are named entities under a catalog, each with
its own settings and optional base location. They are immutable. The
catalog's storageConfigInfo stays the default.
  - Storage mappings let an admin map the name of a namespace or table to a
storage configuration, even before the object exists. Any create under that
name then uses it, including plain Iceberg REST creates and CTAS, with no
client changes.
  - A Polaris create API maps and creates in one call, using the Iceberg
request and response bodies.
  - Each storage configuration can reuse server-side credentials by name
through the existing storageName field from #3409. Credentials stay in
server configuration.
  - Creating a storage configuration only runs static checks, because
access is often granted afterwards. A separate validate call tests access
to its locations on request.

Yufei, mappings differ from the table property you suggested and from the
pending bindings in my last email, so that's the part I'd most like
feedback on. Which cloud identity a table gets is an admin's decision, and
with a property anyone who can create a table would choose it. The PR
spells out the rules for renames, stale mappings and cleanup.

The storageName lookup exists only for AWS today. I'm also working on
separate PRs to add it for GCS and Azure, and the spec assumes them.

Please take a look.

Thanks,
Rishi


On Mon, Oct 26, 2026 at 9:02 PM Yufei Gu <fl...@gmail.com> wrote:

> Thanks, Rishi! This looks good overall.
>
> I'm good with Option B. Catalog scoped storage configurations also make
> sense to me. I agree that having a dedicated entity gives us a cleaner
> model for authorization, versioning, and future extensions.
>
> Regarding the binding issue, I wonder if we can keep things simpler by
> allowing a storage configuration to be specified during table creation(for
> example, specified in a table property, so that engines like Spark don't
> have to adopt a new Polaris-specific plugin), while continuing to inherit
> the parent's configuration by default.
>
> The pending binding approach seems more complicated, especially around
> renames, stale bindings, and cleanup. I'd prefer to avoid introducing that
> complexity unless we have a strong use case that requires it.
>
> I'm good with moving forward with the design doc based on Option B.
>
>
>
> Yufei
>
>
> On Sun, Oct 25, 2026 at 8:49 PM Srinivas Rishindra <sririshindra@gmail.com
> >
> wrote:
>
> > Hi Yufei,
> >
> > Thanks for writing this up. I went through your points and I agree: let's
> > skip Phase 1 and make storage configurations their own entities.
> >
> > On your points:
> >
> >    1. Default base location. This goes beyond create-table. With default
> >    settings a namespace can't have a custom location, so every namespace
> >    location is derived from the catalog's default-base-location. A
> > namespace
> >    bound to an Azure configuration in an S3 catalog would get an s3://
> >    location, and every table in it would fail the allowed-locations
> check.
> > The
> >    only workaround is turning on ALLOW_NAMESPACE_CUSTOM_LOCATION and
> > setting
> >    each namespace's location by hand. An entity doesn't fix this by
> itself
> >    either. I'd give each configuration an optional base location, use it
> > for
> >    namespaces and tables bound before they're created (more below), and
> > refuse
> >    to bind an existing namespace or table whose location isn't inside the
> >    configuration's allowed locations.
> >
> > 2, 3 and 5. Agreed. On 3: #5721 already moved to set/remove fields, but
> > it's
> > still one catalog entity with one version, so two admins editing
> different
> > configurations conflict.
> >
> >    4. Access control. Agreed, and this is the real cost of skipping Phase
> >    1: we deferred AuthZ in June to unblock it, and now we need it up
> front.
> >    I'd follow the policy model: create, read, write, drop and list
> > privileges
> >    on the configuration, and attach as a pair (STORAGE_CONFIG_ATTACH on
> the
> >    configuration plus a dedicated attach privilege on the namespace or
> > table),
> >    all granted to catalog roles. It shouldn't be TABLE_WRITE_PROPERTIES,
> >    because TABLE_WRITE_DATA includes it and every writer could move a
> > table to
> >    another identity. I'd leave out detach. Unlike a policy, unbinding a
> > table
> >    that has data only points it at a configuration that can't reach its
> > files.
> >    Changing a binding is a rebind, with the same location check as above.
> >
> > On your questions:
> >
> >    1.
> >
> >    Skipping Phase 1: +1.
> >    2.
> >
> >    Option B. Policies already do most of what we need, but they fit
> storage
> >    poorly. A policy lives under a namespace, and POLICY_CREATE is a
> > namespace
> >    privilege. Its content is an untyped JSON string. Applicable policies
> > are
> >    returned to anyone with TABLE_READ_PROPERTIES, so role ARNs and
> external
> >    IDs would be visible to every table reader. And resolving them costs a
> >    lookup per path level on every loadTable. I'd copy the patterns
> instead
> >    (attach checked on both sides, versioned updates) into a
> STORAGE_CONFIG
> >    entity, and keep the binding as a configuration name on the namespace
> or
> >    table, as you and Dmitri suggested in June.
> >    3.
> >
> >    Scope. Catalog-scoped. A configuration holds an identity (role ARN and
> >    external ID, Azure tenant, GCS service account) and a fence (allowed
> >    locations, base location). The identity is the part worth sharing. The
> >    fence has to stay per catalog, because the cross-catalog overlap check
> > is
> >    what stops one catalog's admin from getting credentials for another
> >    catalog's files. A configuration shared across catalogs, locations
> >    included, would fail that check, and the only way around it is
> >    ALLOW_OVERLAPPING_CATALOG_URLS, which is realm-wide. So I'd start
> >    catalog-scoped with the identity inline. Later the identity can
> become a
> >    realm-level object, managed by service_admin, that configurations
> > reference
> >    by name. That's additive, and close to how Unity Catalog separates
> >    credentials from external locations.
> >
> > One more thing worth changing while we're at it: how a table ends up on a
> > configuration other than its parent's. A binding needs an existing table,
> > but
> > Polaris writes the first metadata.json (and, for CTAS, vends the first
> > credentials) with the parent's configuration. A Polaris-specific
> > create-and-bind
> > call would get around, but Spark, Trino and dbt can't use it directly.
> >
> > Instead (or in addition), an admin could bind a name ahead of time:
> > "finance.ledger
> > uses configuration finance". When a client then creates finance.ledger
> > through the normal createTable, Polaris uses that configuration for the
> > location, the first write and the credentials, and moves the binding onto
> > the
> > new table. Namespaces work the same way, which is what fixes point 1. The
> > obvious concerns are stale bindings, renames, drop and re-create, and who
> > owns a name that doesn't exist yet. I think each has an answer: a pending
> > binding is used up at create and from then on behaves like any other
> > binding, dropping the configuration deletes its pending bindings, and the
> > namespace owns the names under it, so binding ahead needs the same
> > privileges as binding the namespace. It does need a small table for
> pending
> > bindings. If that's too much, we could allow it for namespaces only;
> tables
> > then follow their namespace.
> >
> > If this sounds right, I'll update the design doc and share it here before
> > opening a spec PR. I'll keep #5721 open until then.
> >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Yufei Gu <fl...@gmail.com>]

Thanks, Rishi! This looks good overall.

I'm good with Option B. Catalog scoped storage configurations also make
sense to me. I agree that having a dedicated entity gives us a cleaner
model for authorization, versioning, and future extensions.

Regarding the binding issue, I wonder if we can keep things simpler by
allowing a storage configuration to be specified during table creation(for
example, specified in a table property, so that engines like Spark don't
have to adopt a new Polaris-specific plugin), while continuing to inherit
the parent's configuration by default.

The pending binding approach seems more complicated, especially around
renames, stale bindings, and cleanup. I'd prefer to avoid introducing that
complexity unless we have a strong use case that requires it.

I'm good with moving forward with the design doc based on Option B.



Yufei


On Sun, Oct 25, 2026 at 8:49 PM Srinivas Rishindra <sr...@gmail.com>
wrote:

> Hi Yufei,
>
> Thanks for writing this up. I went through your points and I agree: let's
> skip Phase 1 and make storage configurations their own entities.
>
> On your points:
>
>    1. Default base location. This goes beyond create-table. With default
>    settings a namespace can't have a custom location, so every namespace
>    location is derived from the catalog's default-base-location. A
> namespace
>    bound to an Azure configuration in an S3 catalog would get an s3://
>    location, and every table in it would fail the allowed-locations check.
> The
>    only workaround is turning on ALLOW_NAMESPACE_CUSTOM_LOCATION and
> setting
>    each namespace's location by hand. An entity doesn't fix this by itself
>    either. I'd give each configuration an optional base location, use it
> for
>    namespaces and tables bound before they're created (more below), and
> refuse
>    to bind an existing namespace or table whose location isn't inside the
>    configuration's allowed locations.
>
> 2, 3 and 5. Agreed. On 3: #5721 already moved to set/remove fields, but
> it's
> still one catalog entity with one version, so two admins editing different
> configurations conflict.
>
>    4. Access control. Agreed, and this is the real cost of skipping Phase
>    1: we deferred AuthZ in June to unblock it, and now we need it up front.
>    I'd follow the policy model: create, read, write, drop and list
> privileges
>    on the configuration, and attach as a pair (STORAGE_CONFIG_ATTACH on the
>    configuration plus a dedicated attach privilege on the namespace or
> table),
>    all granted to catalog roles. It shouldn't be TABLE_WRITE_PROPERTIES,
>    because TABLE_WRITE_DATA includes it and every writer could move a
> table to
>    another identity. I'd leave out detach. Unlike a policy, unbinding a
> table
>    that has data only points it at a configuration that can't reach its
> files.
>    Changing a binding is a rebind, with the same location check as above.
>
> On your questions:
>
>    1.
>
>    Skipping Phase 1: +1.
>    2.
>
>    Option B. Policies already do most of what we need, but they fit storage
>    poorly. A policy lives under a namespace, and POLICY_CREATE is a
> namespace
>    privilege. Its content is an untyped JSON string. Applicable policies
> are
>    returned to anyone with TABLE_READ_PROPERTIES, so role ARNs and external
>    IDs would be visible to every table reader. And resolving them costs a
>    lookup per path level on every loadTable. I'd copy the patterns instead
>    (attach checked on both sides, versioned updates) into a STORAGE_CONFIG
>    entity, and keep the binding as a configuration name on the namespace or
>    table, as you and Dmitri suggested in June.
>    3.
>
>    Scope. Catalog-scoped. A configuration holds an identity (role ARN and
>    external ID, Azure tenant, GCS service account) and a fence (allowed
>    locations, base location). The identity is the part worth sharing. The
>    fence has to stay per catalog, because the cross-catalog overlap check
> is
>    what stops one catalog's admin from getting credentials for another
>    catalog's files. A configuration shared across catalogs, locations
>    included, would fail that check, and the only way around it is
>    ALLOW_OVERLAPPING_CATALOG_URLS, which is realm-wide. So I'd start
>    catalog-scoped with the identity inline. Later the identity can become a
>    realm-level object, managed by service_admin, that configurations
> reference
>    by name. That's additive, and close to how Unity Catalog separates
>    credentials from external locations.
>
> One more thing worth changing while we're at it: how a table ends up on a
> configuration other than its parent's. A binding needs an existing table,
> but
> Polaris writes the first metadata.json (and, for CTAS, vends the first
> credentials) with the parent's configuration. A Polaris-specific
> create-and-bind
> call would get around, but Spark, Trino and dbt can't use it directly.
>
> Instead (or in addition), an admin could bind a name ahead of time:
> "finance.ledger
> uses configuration finance". When a client then creates finance.ledger
> through the normal createTable, Polaris uses that configuration for the
> location, the first write and the credentials, and moves the binding onto
> the
> new table. Namespaces work the same way, which is what fixes point 1. The
> obvious concerns are stale bindings, renames, drop and re-create, and who
> owns a name that doesn't exist yet. I think each has an answer: a pending
> binding is used up at create and from then on behaves like any other
> binding, dropping the configuration deletes its pending bindings, and the
> namespace owns the names under it, so binding ahead needs the same
> privileges as binding the namespace. It does need a small table for pending
> bindings. If that's too much, we could allow it for namespaces only; tables
> then follow their namespace.
>
> If this sounds right, I'll update the design doc and share it here before
> opening a spec PR. I'll keep #5721 open until then.
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Srinivas Rishindra <sr...@gmail.com>]

Hi Yufei,

Thanks for writing this up. I went through your points and I agree: let's
skip Phase 1 and make storage configurations their own entities.

On your points:

   1. Default base location. This goes beyond create-table. With default
   settings a namespace can't have a custom location, so every namespace
   location is derived from the catalog's default-base-location. A namespace
   bound to an Azure configuration in an S3 catalog would get an s3://
   location, and every table in it would fail the allowed-locations check. The
   only workaround is turning on ALLOW_NAMESPACE_CUSTOM_LOCATION and setting
   each namespace's location by hand. An entity doesn't fix this by itself
   either. I'd give each configuration an optional base location, use it for
   namespaces and tables bound before they're created (more below), and refuse
   to bind an existing namespace or table whose location isn't inside the
   configuration's allowed locations.

2, 3 and 5. Agreed. On 3: #5721 already moved to set/remove fields, but it's
still one catalog entity with one version, so two admins editing different
configurations conflict.

   4. Access control. Agreed, and this is the real cost of skipping Phase
   1: we deferred AuthZ in June to unblock it, and now we need it up front.
   I'd follow the policy model: create, read, write, drop and list privileges
   on the configuration, and attach as a pair (STORAGE_CONFIG_ATTACH on the
   configuration plus a dedicated attach privilege on the namespace or table),
   all granted to catalog roles. It shouldn't be TABLE_WRITE_PROPERTIES,
   because TABLE_WRITE_DATA includes it and every writer could move a table to
   another identity. I'd leave out detach. Unlike a policy, unbinding a table
   that has data only points it at a configuration that can't reach its files.
   Changing a binding is a rebind, with the same location check as above.

On your questions:

   1.

   Skipping Phase 1: +1.
   2.

   Option B. Policies already do most of what we need, but they fit storage
   poorly. A policy lives under a namespace, and POLICY_CREATE is a namespace
   privilege. Its content is an untyped JSON string. Applicable policies are
   returned to anyone with TABLE_READ_PROPERTIES, so role ARNs and external
   IDs would be visible to every table reader. And resolving them costs a
   lookup per path level on every loadTable. I'd copy the patterns instead
   (attach checked on both sides, versioned updates) into a STORAGE_CONFIG
   entity, and keep the binding as a configuration name on the namespace or
   table, as you and Dmitri suggested in June.
   3.

   Scope. Catalog-scoped. A configuration holds an identity (role ARN and
   external ID, Azure tenant, GCS service account) and a fence (allowed
   locations, base location). The identity is the part worth sharing. The
   fence has to stay per catalog, because the cross-catalog overlap check is
   what stops one catalog's admin from getting credentials for another
   catalog's files. A configuration shared across catalogs, locations
   included, would fail that check, and the only way around it is
   ALLOW_OVERLAPPING_CATALOG_URLS, which is realm-wide. So I'd start
   catalog-scoped with the identity inline. Later the identity can become a
   realm-level object, managed by service_admin, that configurations reference
   by name. That's additive, and close to how Unity Catalog separates
   credentials from external locations.

One more thing worth changing while we're at it: how a table ends up on a
configuration other than its parent's. A binding needs an existing table, but
Polaris writes the first metadata.json (and, for CTAS, vends the first
credentials) with the parent's configuration. A Polaris-specific
create-and-bind
call would get around, but Spark, Trino and dbt can't use it directly.

Instead (or in addition), an admin could bind a name ahead of time:
"finance.ledger
uses configuration finance". When a client then creates finance.ledger
through the normal createTable, Polaris uses that configuration for the
location, the first write and the credentials, and moves the binding onto the
new table. Namespaces work the same way, which is what fixes point 1. The
obvious concerns are stale bindings, renames, drop and re-create, and who
owns a name that doesn't exist yet. I think each has an answer: a pending
binding is used up at create and from then on behaves like any other
binding, dropping the configuration deletes its pending bindings, and the
namespace owns the names under it, so binding ahead needs the same
privileges as binding the namespace. It does need a small table for pending
bindings. If that's too much, we could allow it for namespaces only; tables
then follow their namespace.

If this sounds right, I'll update the design doc and share it here before
opening a spec PR. I'll keep #5721 open until then.

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Yufei Gu <fl...@gmail.com>]

Hi folks,

While reviewing the storage configuration PR (#5721), we identified several
issues with the Phase 1 design. We think the project would benefit from
skipping Phase 1 and going directly to Phase 2, where storage configuration
becomes a first-class entity. I’d like to hear what the community thinks.

As currently proposed, Phase 1 adds support for multiple storage
configurations within the catalog’s existing StorageConfigInfo. Phase 2
would make storage configuration a separate entity with its own REST
endpoints. Phase 1 was always intended as a transitional step.

Issues we found with Phase 1:

   1.

   *Default base location semantics.* Today, allowedLocations can be empty,
   and the catalog’s default-base-location is added to it automatically.
   With multiple storage configurations, for example, several buckets or a mix
   of S3 and Azure, the catalog default may not apply to every configuration.
   A catalog default pointing to S3 would not work for an Azure configuration.
   Each storage configuration would therefore need its own default base
   location. Without an applicable default, a create-table request that omits
   an explicit location would fail.
   2.

   *Backward compatibility.* Adding a default base location to
   StorageConfigInfo and giving it precedence over the catalog-level
   property would change behavior for existing users, who expect the default
   to come from the catalog properties. Handling this cleanly would require
   deprecation and precedence rules, adding complexity to an interface we
   already plan to replace.
   3.

   *Coarse-grained updates.* Because the configurations live inside the
   catalog, changing a single storage configuration means updating the whole
   catalog object. Supporting targeted updates would require REST endpoints
   similar to those proposed for Phase 2.
   4.

   *No fine-grained access control.* Phase 1 ties storage configuration
   management to catalog administration. Users who need to manage a storage
   configuration would need catalog admin privileges. Separating those
   permissions would require additional design work.
   5.

   *Migration cost and user churn.* Shipping Phase 1 would require us to
   later migrate configurations to the Phase 2 model. Users could see an
   interface introduced, deprecated, and replaced within a short period. We
   expect Phase 1 to require a similar amount of work to Phase 2, with much of
   that work eventually replaced.

We may skip Phase 1 and implement storage configuration directly as a
first-class entity with its own REST API. There are two possible approaches:

   - *Option A: Reuse the policy framework.* A storage configuration could
   be modeled as a new policy type that can be attached at the catalog,
   namespace, and table levels. If the community agrees that storage
   configuration fits the policy model, this may be the simplest path.
   - *Option B: Introduce a dedicated storage configuration entity.* We
   could follow a pattern similar to the Tag proposal (PR #5366), where tags
   live under a catalog and have their own CRUD endpoints. Applied here, users
   would create a catalog and then create one or more storage configurations
   under it.

We also need to decide whether storage configurations should be
catalog-scoped or top-level. Both choices have trade-offs.

I’d especially appreciate feedback on these questions:

   1. Are there objections to skipping Phase 1?
   2. Do you prefer Option A (reuse the policy framework) or Option B (a
   dedicated entity)?
   3. Should storage configurations be catalog-scoped or top-level?

Yufei


On Tue, Oct 20, 2026 at 2:33 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> Srinivas opened PR [5721] for the Management API change related to
> supporting multiple storage configurations per catalog.
>
> Please take a look.
>
> Thanks,
> Dmitri.
>
> [5721] https://github.com/apache/polaris/pull/5721
>
> On Fri, Aug 28, 2026 at 12:35 PM Alexandre Dutra <ad...@apache.org>
> wrote:
>
> > Hi Srinivas, hi all,
> >
> > As Dmitri said, summer time indeed has slowed down the community
> > bandwidth, but I'm happy to get back to this topic today.
> >
> > Thanks for the thorough writeup and also the summary of the current
> > status quo: very useful. I'm OK with both the two-phase framing and
> > the fail-closed approach.
> >
> > I have however one concern on Phase 1 that I think is worth settling
> > before we move forward: the storageName field is being **overloaded to
> > do two independent jobs**.
> >
> > Today (per #3409) this field has exactly one purpose: select the
> > deployment-time credential set. Phase 1 quietly adds a second: it
> > becomes the identity key of the named-config map and the target of
> > polaris.storage.name. Those are two different namespaces: a
> > catalog-scoped config identity vs. a deployment-scoped credential
> > selector; welding them together means you can't share one credential
> > set across configs, and you can't rename a config without re-pointing
> > its credentials.
> >
> > This surfaces as two concrete rough edges:
> >
> > 1. The default config's storageName is ambiguous. §5.1 excludes the
> > default's name from the named map, but §5.4 resolves references only
> > against that map. So if the default carries storageName = "foo" and a
> > table sets polaris.storage.name = "foo", the lookup misses and we fail
> > closed with a "dangling reference", even though a config named "foo"
> > plainly exists. **The default config having a name at all is a code
> > smell.**
> >
> > 2. The two usages can disagree. If named references are found in the
> > hierarchy, but RESOLVE_CREDENTIALS_BY_STORAGE_NAME is off, we'd
> > resolve a table to a named storage config, but wouldn't pick named
> > credentials for it, falling back to the default provider chain. That's
> > a silent fail-open path inside a design tailored for fail-closed.
> >
> > My proposal: decouple the two concepts, and use a real map of name ->
> > StorageConfigInfo in Create/UpdateCatalogRequest; credential selection
> > keeps using the current storageName field (we could maybe rename it).
> > The map key becomes the reference identity; uniqueness comes for free.
> > That fully decouples the two concerns: several configs can share one
> > credential set, and the default no longer needs a name.
> >
> > WDYT?
> >
> > Thanks,
> > Alex
> >
> > On Mon, Aug 10, 2026 at 7:04 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> > >
> > > Hi Srinivas,
> > >
> > > Thanks for putting this document together. I think it provides an
> > excellent
> > > summary of this discussion and lays out clean path forward.
> > >
> > > As for phase 1 it looks like we could start implementing it, if other
> > > reviews agree. Given that it's summer time it may take a week or two
> for
> > > everybody to catch up on this thread, though.
> > >
> > > From a practical point of view, I tend to think that non-default
> storage
> > > config can be use effectively only at the namespace level. For a new
> > table
> > > to be created via the IRC API, the request can only operate on the
> > > name/config stored in the immediate namespace or higher. The table
> itself
> > > cannot define the storage name unless we revisit the point of managing
> it
> > > only via Polaris properties.
> > >
> > > It might be worth allowing Iceberg table properties to carry the
> storage
> > > name too. Polaris can forward that information to its internal
> properties
> > > on changes. However, this point does not affect Phase 1, I think. We
> can
> > > add this kind of property forwarding later.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Wed, Aug 5, 2026 at 10:51 AM Srinivas Rishindra <
> > sririshindra@gmail.com>
> > > wrote:
> > >
> > > > I've put together an updated design proposal document detailing the
> > storage
> > > > configuration management proposal. Please provide your feedback on
> the
> > > > proposal.
> > > >
> > > >
> > > >
> >
> https://docs.google.com/document/d/1TSaZ0G2oEoWq6lDMSPIIfHLwXxW9VZmhUNE3s_jfxgI/edit?usp=sharing
> > > >
> > > >
> > > >
> > > > On Thu, Jun 18, 2026 at 3:23 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Adam,
> > > > >
> > > > > Could you give some more details, please, about where you think the
> > > > Iceberg
> > > > > spec disallows multiple storage per table?
> > > > >
> > > > > I never thought it was not allowed. Perhaps it was inconvenient or
> > > > > impractical in existing clients, but the spec itself seems to be
> > fairly
> > > > > unbiased with respect to actual storage technology... Perhaps I
> > missed
> > > > > something, though.
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <sz...@apache.org>
> wrote:
> > > > >
> > > > > > Hi Yufei,
> > > > > >
> > > > > > Regarding #2:
> > > > > > Would opting for a simple (assuming per table) storageConfigName
> or
> > > > > > storageConfigId
> > > > > > support future extensibility?
> > > > > > Asking from a multiple-storage-per-table perspective as it's
> > something
> > > > > that
> > > > > > the Iceberg spec does allow IMHO.
> > > > > >
> > > > > > Cheers,
> > > > > > Adam
> > > > > >
> > > > > > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <fl...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi folks, thanks for the discussion. They are really helpful!
> > > > > > >
> > > > > > > I think there are two separate questions here:
> > > > > > >
> > > > > > > 1. How should we store storage configurations? As catalog
> > properties
> > > > or
> > > > > > as
> > > > > > > separate entities?
> > > > > > > I'd lean toward separate entities in the long term. It
> provides a
> > > > > cleaner
> > > > > > > boundary between a catalog and its storage configurations,
> avoids
> > > > > > bloating
> > > > > > > the catalog object as the number of configs grows, and gives
> us a
> > > > > better
> > > > > > > foundation for future access control since entity level
> > permissions
> > > > are
> > > > > > > easier to reason about.
> > > > > > >
> > > > > > > 2. How should a table or namespace refer to a storage
> > configuration?
> > > > > > > I'd prefer storing a simple reference on the entity itself
> rather
> > > > than
> > > > > > > introducing a separate relationship table. Both approaches
> > support
> > > > > > > hierarchical lookup and inheritance when a storage config is
> not
> > > > > > specified
> > > > > > > at the current level. However, I don't think we should allow
> > multiple
> > > > > > > storage configs to be associated with a single table or
> > namespace.
> > > > > Given
> > > > > > > that cardinality, a simple storageConfigName or storageConfigId
> > field
> > > > > > seems
> > > > > > > sufficient and easier to understand than a dedicated
> relationship
> > > > > model.
> > > > > > >
> > > > > > > For example:
> > > > > > >
> > > > > > > Catalog
> > > > > > > ├─ entity: StorageConfigA
> > > > > > > ├─ entity: StorageConfigB
> > > > > > > └─ entity: StorageConfigC
> > > > > > >
> > > > > > > Table1 storage property -> StorageConfigA
> > > > > > > Table2 storage property -> StorageConfigB
> > > > > > > Table3 storage property -> StorageConfigC
> > > > > > >
> > > > > > > To me, this keeps the model simple while still supporting
> shared
> > > > > storage
> > > > > > > configurations and future extensibility.
> > > > > > >
> > > > > > > Yufei
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Robert,
> > > > > > > >
> > > > > > > > The distinction between "attached" and "referenced" config
> > makes
> > > > > sense.
> > > > > > > >
> > > > > > > > My impression from JB's message on Mar 19 in this thread [1]
> is
> > > > that
> > > > > JB
> > > > > > > is
> > > > > > > > specifically proposing the "referenced" model, though.
> > > > > > > >
> > > > > > > > From my POV, both approaches can work, but the "referenced"
> > model
> > > > > might
> > > > > > > be
> > > > > > > > more intuitive to users.
> > > > > > > >
> > > > > > > > [1]
> > > > https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <snazy@snazy.de
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > AFAIK the current Polaris code appears to resolve the
> storage
> > > > > > > > > configuration from the leaf (table), upwards via the
> > namespace
> > > > > > elements
> > > > > > > > to
> > > > > > > > > the catalog, choosing the first one found.
> > > > > > > > > I am not sure whether that functionality is already used
> > > > anywhere?
> > > > > > > > > Being able to use a storage configuration on a namespace
> or a
> > > > leaf
> > > > > > > > appears
> > > > > > > > > appealing.
> > > > > > > > >
> > > > > > > > > OTOH I see a demand for named storage configurations.
> > > > > > > > >
> > > > > > > > > I wonder whether the mechanism to derive the storage
> > > > configuration
> > > > > > > could
> > > > > > > > > support both "directly attached" and "referenced" storage
> > > > > > > configurations:
> > > > > > > > > A namespace element or leaf can have either a "directly
> > attached"
> > > > > > > storage
> > > > > > > > > configuration or a reference to a named one.
> > > > > > > > >
> > > > > > > > > We'd need management-ish APIs to handle the storage
> > > > configurations
> > > > > > for
> > > > > > > a
> > > > > > > > > namespace or leaf.
> > > > > > > > > These APIs, which do not exist yet, could start with
> > "directly
> > > > > > > attached"
> > > > > > > > > storage configurations and once the authz questions are
> > resolved,
> > > > > > > extend
> > > > > > > > to
> > > > > > > > > include named, shared storage configurations.
> > > > > > > > >
> > > > > > > > > Thoughts?
> > > > > > > > >
> > > > > > > > > Robert
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi All,
> > > > > > > > > >
> > > > > > > > > > To recap the discussion from the Community Sync today: My
> > > > > > > understanding
> > > > > > > > > is
> > > > > > > > > > that the consensus was:
> > > > > > > > > >
> > > > > > > > > > * Defer AuthZ aspects for later discussion.
> > > > > > > > > >
> > > > > > > > > > * Allow multiple Storage Config objects per Catalog. If
> > more
> > > > than
> > > > > > one
> > > > > > > > > > Storage Config is present, they are to have different
> > (within
> > > > the
> > > > > > > > > catalog)
> > > > > > > > > > names. Rishi to open a new PR for this (scoped to REST
> API
> > +
> > > > > Sever
> > > > > > > > > > changes).
> > > > > > > > > >
> > > > > > > > > > * Defer CLI changes until we have the full end-to-end
> > system
> > > > > > working
> > > > > > > > (it
> > > > > > > > > > should be usable via curl)
> > > > > > > > > >
> > > > > > > > > > * PR 4023 is on hold for now. It is to be rebased after
> > adding
> > > > > > > support
> > > > > > > > > for
> > > > > > > > > > multiple Storage Configs
> > > > > > > > > >
> > > > > > > > > > * The connection between storage names and storage access
> > > > > > credentials
> > > > > > > > is
> > > > > > > > > to
> > > > > > > > > > remain unchanged for now. It is effectively a deployment
> > time
> > > > > > concern
> > > > > > > > > given
> > > > > > > > > > the current OSS code.
> > > > > > > > > >
> > > > > > > > > > Please correct / add if I missed anything.
> > > > > > > > > >
> > > > > > > > > > From my POV, I think we still need to discuss how exactly
> > > > > entities
> > > > > > > > > (tables,
> > > > > > > > > > views) will link to Storage Config before we can resume
> PR
> > > > 4023.
> > > > > > > > > >
> > > > > > > > > > I propose storing the config name in the Entity
> properties
> > (not
> > > > > in
> > > > > > > > Table
> > > > > > > > > > Metadata properties).
> > > > > > > > > >
> > > > > > > > > > If clients also need a way to define it while interacting
> > with
> > > > > > > Polaris
> > > > > > > > > over
> > > > > > > > > > the IRC API, let's discuss how that can be done.
> > > > > > > > > >
> > > > > > > > > > WDYT?
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Srinivas,
> > > > > > > > > > >
> > > > > > > > > > > I see  that PR 4023 is active again. I'll try and post
> a
> > > > fresh
> > > > > > > review
> > > > > > > > > > soon.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > > > > > > > > > sririshindra@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi JB and Dmitri,
> > > > > > > > > > >>
> > > > > > > > > > >> I raised PR #4023 <
> > > > > https://github.com/apache/polaris/pull/4023>
> > > > > > > > > > yesterday
> > > > > > > > > > >> as part of the broader per-table storage config
> effort,
> > and
> > > > I
> > > > > > > > believe
> > > > > > > > > it
> > > > > > > > > > >> implements exactly what you are proposing.
> > > > > > > > > > >>
> > > > > > > > > > >> Building on the named storage configs introduced in PR
> > #3409
> > > > > > > > > > >> <https://github.com/apache/polaris/pull/3409>, this
> PR
> > > > allows
> > > > > > > > > > namespaces
> > > > > > > > > > >> and tables to specify a polaris.storage.name
> property.
> > This
> > > > > > acts
> > > > > > > as
> > > > > > > > > the
> > > > > > > > > > >> "identifier" JB mentioned, allowing credential vending
> > to
> > > > > > resolve
> > > > > > > > the
> > > > > > > > > > >> correct, named storage config at runtime without
> > introducing
> > > > > new
> > > > > > > > > > >> management
> > > > > > > > > > >> APIs.
> > > > > > > > > > >>
> > > > > > > > > > >> Could you please take a look at the PR and let me know
> > if
> > > > this
> > > > > > > > aligns
> > > > > > > > > > with
> > > > > > > > > > >> what you are looking for?
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

Srinivas opened PR [5721] for the Management API change related to
supporting multiple storage configurations per catalog.

Please take a look.

Thanks,
Dmitri.

[5721] https://github.com/apache/polaris/pull/5721

On Fri, Aug 28, 2026 at 12:35 PM Alexandre Dutra <ad...@apache.org> wrote:

> Hi Srinivas, hi all,
>
> As Dmitri said, summer time indeed has slowed down the community
> bandwidth, but I'm happy to get back to this topic today.
>
> Thanks for the thorough writeup and also the summary of the current
> status quo: very useful. I'm OK with both the two-phase framing and
> the fail-closed approach.
>
> I have however one concern on Phase 1 that I think is worth settling
> before we move forward: the storageName field is being **overloaded to
> do two independent jobs**.
>
> Today (per #3409) this field has exactly one purpose: select the
> deployment-time credential set. Phase 1 quietly adds a second: it
> becomes the identity key of the named-config map and the target of
> polaris.storage.name. Those are two different namespaces: a
> catalog-scoped config identity vs. a deployment-scoped credential
> selector; welding them together means you can't share one credential
> set across configs, and you can't rename a config without re-pointing
> its credentials.
>
> This surfaces as two concrete rough edges:
>
> 1. The default config's storageName is ambiguous. §5.1 excludes the
> default's name from the named map, but §5.4 resolves references only
> against that map. So if the default carries storageName = "foo" and a
> table sets polaris.storage.name = "foo", the lookup misses and we fail
> closed with a "dangling reference", even though a config named "foo"
> plainly exists. **The default config having a name at all is a code
> smell.**
>
> 2. The two usages can disagree. If named references are found in the
> hierarchy, but RESOLVE_CREDENTIALS_BY_STORAGE_NAME is off, we'd
> resolve a table to a named storage config, but wouldn't pick named
> credentials for it, falling back to the default provider chain. That's
> a silent fail-open path inside a design tailored for fail-closed.
>
> My proposal: decouple the two concepts, and use a real map of name ->
> StorageConfigInfo in Create/UpdateCatalogRequest; credential selection
> keeps using the current storageName field (we could maybe rename it).
> The map key becomes the reference identity; uniqueness comes for free.
> That fully decouples the two concerns: several configs can share one
> credential set, and the default no longer needs a name.
>
> WDYT?
>
> Thanks,
> Alex
>
> On Mon, Aug 10, 2026 at 7:04 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
> >
> > Hi Srinivas,
> >
> > Thanks for putting this document together. I think it provides an
> excellent
> > summary of this discussion and lays out clean path forward.
> >
> > As for phase 1 it looks like we could start implementing it, if other
> > reviews agree. Given that it's summer time it may take a week or two for
> > everybody to catch up on this thread, though.
> >
> > From a practical point of view, I tend to think that non-default storage
> > config can be use effectively only at the namespace level. For a new
> table
> > to be created via the IRC API, the request can only operate on the
> > name/config stored in the immediate namespace or higher. The table itself
> > cannot define the storage name unless we revisit the point of managing it
> > only via Polaris properties.
> >
> > It might be worth allowing Iceberg table properties to carry the storage
> > name too. Polaris can forward that information to its internal properties
> > on changes. However, this point does not affect Phase 1, I think. We can
> > add this kind of property forwarding later.
> >
> > Cheers,
> > Dmitri.
> >
> > On Wed, Aug 5, 2026 at 10:51 AM Srinivas Rishindra <
> sririshindra@gmail.com>
> > wrote:
> >
> > > I've put together an updated design proposal document detailing the
> storage
> > > configuration management proposal. Please provide your feedback on the
> > > proposal.
> > >
> > >
> > >
> https://docs.google.com/document/d/1TSaZ0G2oEoWq6lDMSPIIfHLwXxW9VZmhUNE3s_jfxgI/edit?usp=sharing
> > >
> > >
> > >
> > > On Thu, Jun 18, 2026 at 3:23 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Adam,
> > > >
> > > > Could you give some more details, please, about where you think the
> > > Iceberg
> > > > spec disallows multiple storage per table?
> > > >
> > > > I never thought it was not allowed. Perhaps it was inconvenient or
> > > > impractical in existing clients, but the spec itself seems to be
> fairly
> > > > unbiased with respect to actual storage technology... Perhaps I
> missed
> > > > something, though.
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <sz...@apache.org> wrote:
> > > >
> > > > > Hi Yufei,
> > > > >
> > > > > Regarding #2:
> > > > > Would opting for a simple (assuming per table) storageConfigName or
> > > > > storageConfigId
> > > > > support future extensibility?
> > > > > Asking from a multiple-storage-per-table perspective as it's
> something
> > > > that
> > > > > the Iceberg spec does allow IMHO.
> > > > >
> > > > > Cheers,
> > > > > Adam
> > > > >
> > > > > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <fl...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi folks, thanks for the discussion. They are really helpful!
> > > > > >
> > > > > > I think there are two separate questions here:
> > > > > >
> > > > > > 1. How should we store storage configurations? As catalog
> properties
> > > or
> > > > > as
> > > > > > separate entities?
> > > > > > I'd lean toward separate entities in the long term. It provides a
> > > > cleaner
> > > > > > boundary between a catalog and its storage configurations, avoids
> > > > > bloating
> > > > > > the catalog object as the number of configs grows, and gives us a
> > > > better
> > > > > > foundation for future access control since entity level
> permissions
> > > are
> > > > > > easier to reason about.
> > > > > >
> > > > > > 2. How should a table or namespace refer to a storage
> configuration?
> > > > > > I'd prefer storing a simple reference on the entity itself rather
> > > than
> > > > > > introducing a separate relationship table. Both approaches
> support
> > > > > > hierarchical lookup and inheritance when a storage config is not
> > > > > specified
> > > > > > at the current level. However, I don't think we should allow
> multiple
> > > > > > storage configs to be associated with a single table or
> namespace.
> > > > Given
> > > > > > that cardinality, a simple storageConfigName or storageConfigId
> field
> > > > > seems
> > > > > > sufficient and easier to understand than a dedicated relationship
> > > > model.
> > > > > >
> > > > > > For example:
> > > > > >
> > > > > > Catalog
> > > > > > ├─ entity: StorageConfigA
> > > > > > ├─ entity: StorageConfigB
> > > > > > └─ entity: StorageConfigC
> > > > > >
> > > > > > Table1 storage property -> StorageConfigA
> > > > > > Table2 storage property -> StorageConfigB
> > > > > > Table3 storage property -> StorageConfigC
> > > > > >
> > > > > > To me, this keeps the model simple while still supporting shared
> > > > storage
> > > > > > configurations and future extensibility.
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Robert,
> > > > > > >
> > > > > > > The distinction between "attached" and "referenced" config
> makes
> > > > sense.
> > > > > > >
> > > > > > > My impression from JB's message on Mar 19 in this thread [1] is
> > > that
> > > > JB
> > > > > > is
> > > > > > > specifically proposing the "referenced" model, though.
> > > > > > >
> > > > > > > From my POV, both approaches can work, but the "referenced"
> model
> > > > might
> > > > > > be
> > > > > > > more intuitive to users.
> > > > > > >
> > > > > > > [1]
> > > https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <sn...@snazy.de>
> > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > AFAIK the current Polaris code appears to resolve the storage
> > > > > > > > configuration from the leaf (table), upwards via the
> namespace
> > > > > elements
> > > > > > > to
> > > > > > > > the catalog, choosing the first one found.
> > > > > > > > I am not sure whether that functionality is already used
> > > anywhere?
> > > > > > > > Being able to use a storage configuration on a namespace or a
> > > leaf
> > > > > > > appears
> > > > > > > > appealing.
> > > > > > > >
> > > > > > > > OTOH I see a demand for named storage configurations.
> > > > > > > >
> > > > > > > > I wonder whether the mechanism to derive the storage
> > > configuration
> > > > > > could
> > > > > > > > support both "directly attached" and "referenced" storage
> > > > > > configurations:
> > > > > > > > A namespace element or leaf can have either a "directly
> attached"
> > > > > > storage
> > > > > > > > configuration or a reference to a named one.
> > > > > > > >
> > > > > > > > We'd need management-ish APIs to handle the storage
> > > configurations
> > > > > for
> > > > > > a
> > > > > > > > namespace or leaf.
> > > > > > > > These APIs, which do not exist yet, could start with
> "directly
> > > > > > attached"
> > > > > > > > storage configurations and once the authz questions are
> resolved,
> > > > > > extend
> > > > > > > to
> > > > > > > > include named, shared storage configurations.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > Robert
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > To recap the discussion from the Community Sync today: My
> > > > > > understanding
> > > > > > > > is
> > > > > > > > > that the consensus was:
> > > > > > > > >
> > > > > > > > > * Defer AuthZ aspects for later discussion.
> > > > > > > > >
> > > > > > > > > * Allow multiple Storage Config objects per Catalog. If
> more
> > > than
> > > > > one
> > > > > > > > > Storage Config is present, they are to have different
> (within
> > > the
> > > > > > > > catalog)
> > > > > > > > > names. Rishi to open a new PR for this (scoped to REST API
> +
> > > > Sever
> > > > > > > > > changes).
> > > > > > > > >
> > > > > > > > > * Defer CLI changes until we have the full end-to-end
> system
> > > > > working
> > > > > > > (it
> > > > > > > > > should be usable via curl)
> > > > > > > > >
> > > > > > > > > * PR 4023 is on hold for now. It is to be rebased after
> adding
> > > > > > support
> > > > > > > > for
> > > > > > > > > multiple Storage Configs
> > > > > > > > >
> > > > > > > > > * The connection between storage names and storage access
> > > > > credentials
> > > > > > > is
> > > > > > > > to
> > > > > > > > > remain unchanged for now. It is effectively a deployment
> time
> > > > > concern
> > > > > > > > given
> > > > > > > > > the current OSS code.
> > > > > > > > >
> > > > > > > > > Please correct / add if I missed anything.
> > > > > > > > >
> > > > > > > > > From my POV, I think we still need to discuss how exactly
> > > > entities
> > > > > > > > (tables,
> > > > > > > > > views) will link to Storage Config before we can resume PR
> > > 4023.
> > > > > > > > >
> > > > > > > > > I propose storing the config name in the Entity properties
> (not
> > > > in
> > > > > > > Table
> > > > > > > > > Metadata properties).
> > > > > > > > >
> > > > > > > > > If clients also need a way to define it while interacting
> with
> > > > > > Polaris
> > > > > > > > over
> > > > > > > > > the IRC API, let's discuss how that can be done.
> > > > > > > > >
> > > > > > > > > WDYT?
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Srinivas,
> > > > > > > > > >
> > > > > > > > > > I see  that PR 4023 is active again. I'll try and post a
> > > fresh
> > > > > > review
> > > > > > > > > soon.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > > > > > > > > sririshindra@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi JB and Dmitri,
> > > > > > > > > >>
> > > > > > > > > >> I raised PR #4023 <
> > > > https://github.com/apache/polaris/pull/4023>
> > > > > > > > > yesterday
> > > > > > > > > >> as part of the broader per-table storage config effort,
> and
> > > I
> > > > > > > believe
> > > > > > > > it
> > > > > > > > > >> implements exactly what you are proposing.
> > > > > > > > > >>
> > > > > > > > > >> Building on the named storage configs introduced in PR
> #3409
> > > > > > > > > >> <https://github.com/apache/polaris/pull/3409>, this PR
> > > allows
> > > > > > > > > namespaces
> > > > > > > > > >> and tables to specify a polaris.storage.name property.
> This
> > > > > acts
> > > > > > as
> > > > > > > > the
> > > > > > > > > >> "identifier" JB mentioned, allowing credential vending
> to
> > > > > resolve
> > > > > > > the
> > > > > > > > > >> correct, named storage config at runtime without
> introducing
> > > > new
> > > > > > > > > >> management
> > > > > > > > > >> APIs.
> > > > > > > > > >>
> > > > > > > > > >> Could you please take a look at the PR and let me know
> if
> > > this
> > > > > > > aligns
> > > > > > > > > with
> > > > > > > > > >> what you are looking for?
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Srinivas Rishindra <sr...@gmail.com>]

Thanks Yufei,
Here is the first PR https://github.com/apache/polaris/pull/5556

On Tue, Sep 8, 2026 at 5:03 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi Srinivas, Thanks for putting together the design doc and moving
> this forward. The design doc looks good to me overall. Looking forward
> to the PRs!
>
> Yufei
>
> On Fri, Aug 28, 2026 at 9:35 AM Alexandre Dutra <ad...@apache.org> wrote:
> >
> > Hi Srinivas, hi all,
> >
> > As Dmitri said, summer time indeed has slowed down the community
> > bandwidth, but I'm happy to get back to this topic today.
> >
> > Thanks for the thorough writeup and also the summary of the current
> > status quo: very useful. I'm OK with both the two-phase framing and
> > the fail-closed approach.
> >
> > I have however one concern on Phase 1 that I think is worth settling
> > before we move forward: the storageName field is being **overloaded to
> > do two independent jobs**.
> >
> > Today (per #3409) this field has exactly one purpose: select the
> > deployment-time credential set. Phase 1 quietly adds a second: it
> > becomes the identity key of the named-config map and the target of
> > polaris.storage.name. Those are two different namespaces: a
> > catalog-scoped config identity vs. a deployment-scoped credential
> > selector; welding them together means you can't share one credential
> > set across configs, and you can't rename a config without re-pointing
> > its credentials.
> >
> > This surfaces as two concrete rough edges:
> >
> > 1. The default config's storageName is ambiguous. §5.1 excludes the
> > default's name from the named map, but §5.4 resolves references only
> > against that map. So if the default carries storageName = "foo" and a
> > table sets polaris.storage.name = "foo", the lookup misses and we fail
> > closed with a "dangling reference", even though a config named "foo"
> > plainly exists. **The default config having a name at all is a code
> > smell.**
> >
> > 2. The two usages can disagree. If named references are found in the
> > hierarchy, but RESOLVE_CREDENTIALS_BY_STORAGE_NAME is off, we'd
> > resolve a table to a named storage config, but wouldn't pick named
> > credentials for it, falling back to the default provider chain. That's
> > a silent fail-open path inside a design tailored for fail-closed.
> >
> > My proposal: decouple the two concepts, and use a real map of name ->
> > StorageConfigInfo in Create/UpdateCatalogRequest; credential selection
> > keeps using the current storageName field (we could maybe rename it).
> > The map key becomes the reference identity; uniqueness comes for free.
> > That fully decouples the two concerns: several configs can share one
> > credential set, and the default no longer needs a name.
> >
> > WDYT?
> >
> > Thanks,
> > Alex
> >
> > On Mon, Aug 10, 2026 at 7:04 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
> > >
> > > Hi Srinivas,
> > >
> > > Thanks for putting this document together. I think it provides an
> excellent
> > > summary of this discussion and lays out clean path forward.
> > >
> > > As for phase 1 it looks like we could start implementing it, if other
> > > reviews agree. Given that it's summer time it may take a week or two
> for
> > > everybody to catch up on this thread, though.
> > >
> > > From a practical point of view, I tend to think that non-default
> storage
> > > config can be use effectively only at the namespace level. For a new
> table
> > > to be created via the IRC API, the request can only operate on the
> > > name/config stored in the immediate namespace or higher. The table
> itself
> > > cannot define the storage name unless we revisit the point of managing
> it
> > > only via Polaris properties.
> > >
> > > It might be worth allowing Iceberg table properties to carry the
> storage
> > > name too. Polaris can forward that information to its internal
> properties
> > > on changes. However, this point does not affect Phase 1, I think. We
> can
> > > add this kind of property forwarding later.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Wed, Aug 5, 2026 at 10:51 AM Srinivas Rishindra <
> sririshindra@gmail.com>
> > > wrote:
> > >
> > > > I've put together an updated design proposal document detailing the
> storage
> > > > configuration management proposal. Please provide your feedback on
> the
> > > > proposal.
> > > >
> > > >
> > > >
> https://docs.google.com/document/d/1TSaZ0G2oEoWq6lDMSPIIfHLwXxW9VZmhUNE3s_jfxgI/edit?usp=sharing
> > > >
> > > >
> > > >
> > > > On Thu, Jun 18, 2026 at 3:23 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Adam,
> > > > >
> > > > > Could you give some more details, please, about where you think the
> > > > Iceberg
> > > > > spec disallows multiple storage per table?
> > > > >
> > > > > I never thought it was not allowed. Perhaps it was inconvenient or
> > > > > impractical in existing clients, but the spec itself seems to be
> fairly
> > > > > unbiased with respect to actual storage technology... Perhaps I
> missed
> > > > > something, though.
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <sz...@apache.org>
> wrote:
> > > > >
> > > > > > Hi Yufei,
> > > > > >
> > > > > > Regarding #2:
> > > > > > Would opting for a simple (assuming per table) storageConfigName
> or
> > > > > > storageConfigId
> > > > > > support future extensibility?
> > > > > > Asking from a multiple-storage-per-table perspective as it's
> something
> > > > > that
> > > > > > the Iceberg spec does allow IMHO.
> > > > > >
> > > > > > Cheers,
> > > > > > Adam
> > > > > >
> > > > > > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <fl...@gmail.com>
> wrote:
> > > > > >
> > > > > > > Hi folks, thanks for the discussion. They are really helpful!
> > > > > > >
> > > > > > > I think there are two separate questions here:
> > > > > > >
> > > > > > > 1. How should we store storage configurations? As catalog
> properties
> > > > or
> > > > > > as
> > > > > > > separate entities?
> > > > > > > I'd lean toward separate entities in the long term. It
> provides a
> > > > > cleaner
> > > > > > > boundary between a catalog and its storage configurations,
> avoids
> > > > > > bloating
> > > > > > > the catalog object as the number of configs grows, and gives
> us a
> > > > > better
> > > > > > > foundation for future access control since entity level
> permissions
> > > > are
> > > > > > > easier to reason about.
> > > > > > >
> > > > > > > 2. How should a table or namespace refer to a storage
> configuration?
> > > > > > > I'd prefer storing a simple reference on the entity itself
> rather
> > > > than
> > > > > > > introducing a separate relationship table. Both approaches
> support
> > > > > > > hierarchical lookup and inheritance when a storage config is
> not
> > > > > > specified
> > > > > > > at the current level. However, I don't think we should allow
> multiple
> > > > > > > storage configs to be associated with a single table or
> namespace.
> > > > > Given
> > > > > > > that cardinality, a simple storageConfigName or
> storageConfigId field
> > > > > > seems
> > > > > > > sufficient and easier to understand than a dedicated
> relationship
> > > > > model.
> > > > > > >
> > > > > > > For example:
> > > > > > >
> > > > > > > Catalog
> > > > > > > ├─ entity: StorageConfigA
> > > > > > > ├─ entity: StorageConfigB
> > > > > > > └─ entity: StorageConfigC
> > > > > > >
> > > > > > > Table1 storage property -> StorageConfigA
> > > > > > > Table2 storage property -> StorageConfigB
> > > > > > > Table3 storage property -> StorageConfigC
> > > > > > >
> > > > > > > To me, this keeps the model simple while still supporting
> shared
> > > > > storage
> > > > > > > configurations and future extensibility.
> > > > > > >
> > > > > > > Yufei
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Robert,
> > > > > > > >
> > > > > > > > The distinction between "attached" and "referenced" config
> makes
> > > > > sense.
> > > > > > > >
> > > > > > > > My impression from JB's message on Mar 19 in this thread [1]
> is
> > > > that
> > > > > JB
> > > > > > > is
> > > > > > > > specifically proposing the "referenced" model, though.
> > > > > > > >
> > > > > > > > From my POV, both approaches can work, but the "referenced"
> model
> > > > > might
> > > > > > > be
> > > > > > > > more intuitive to users.
> > > > > > > >
> > > > > > > > [1]
> > > > https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <snazy@snazy.de
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > AFAIK the current Polaris code appears to resolve the
> storage
> > > > > > > > > configuration from the leaf (table), upwards via the
> namespace
> > > > > > elements
> > > > > > > > to
> > > > > > > > > the catalog, choosing the first one found.
> > > > > > > > > I am not sure whether that functionality is already used
> > > > anywhere?
> > > > > > > > > Being able to use a storage configuration on a namespace
> or a
> > > > leaf
> > > > > > > > appears
> > > > > > > > > appealing.
> > > > > > > > >
> > > > > > > > > OTOH I see a demand for named storage configurations.
> > > > > > > > >
> > > > > > > > > I wonder whether the mechanism to derive the storage
> > > > configuration
> > > > > > > could
> > > > > > > > > support both "directly attached" and "referenced" storage
> > > > > > > configurations:
> > > > > > > > > A namespace element or leaf can have either a "directly
> attached"
> > > > > > > storage
> > > > > > > > > configuration or a reference to a named one.
> > > > > > > > >
> > > > > > > > > We'd need management-ish APIs to handle the storage
> > > > configurations
> > > > > > for
> > > > > > > a
> > > > > > > > > namespace or leaf.
> > > > > > > > > These APIs, which do not exist yet, could start with
> "directly
> > > > > > > attached"
> > > > > > > > > storage configurations and once the authz questions are
> resolved,
> > > > > > > extend
> > > > > > > > to
> > > > > > > > > include named, shared storage configurations.
> > > > > > > > >
> > > > > > > > > Thoughts?
> > > > > > > > >
> > > > > > > > > Robert
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi All,
> > > > > > > > > >
> > > > > > > > > > To recap the discussion from the Community Sync today: My
> > > > > > > understanding
> > > > > > > > > is
> > > > > > > > > > that the consensus was:
> > > > > > > > > >
> > > > > > > > > > * Defer AuthZ aspects for later discussion.
> > > > > > > > > >
> > > > > > > > > > * Allow multiple Storage Config objects per Catalog. If
> more
> > > > than
> > > > > > one
> > > > > > > > > > Storage Config is present, they are to have different
> (within
> > > > the
> > > > > > > > > catalog)
> > > > > > > > > > names. Rishi to open a new PR for this (scoped to REST
> API +
> > > > > Sever
> > > > > > > > > > changes).
> > > > > > > > > >
> > > > > > > > > > * Defer CLI changes until we have the full end-to-end
> system
> > > > > > working
> > > > > > > > (it
> > > > > > > > > > should be usable via curl)
> > > > > > > > > >
> > > > > > > > > > * PR 4023 is on hold for now. It is to be rebased after
> adding
> > > > > > > support
> > > > > > > > > for
> > > > > > > > > > multiple Storage Configs
> > > > > > > > > >
> > > > > > > > > > * The connection between storage names and storage access
> > > > > > credentials
> > > > > > > > is
> > > > > > > > > to
> > > > > > > > > > remain unchanged for now. It is effectively a deployment
> time
> > > > > > concern
> > > > > > > > > given
> > > > > > > > > > the current OSS code.
> > > > > > > > > >
> > > > > > > > > > Please correct / add if I missed anything.
> > > > > > > > > >
> > > > > > > > > > From my POV, I think we still need to discuss how exactly
> > > > > entities
> > > > > > > > > (tables,
> > > > > > > > > > views) will link to Storage Config before we can resume
> PR
> > > > 4023.
> > > > > > > > > >
> > > > > > > > > > I propose storing the config name in the Entity
> properties (not
> > > > > in
> > > > > > > > Table
> > > > > > > > > > Metadata properties).
> > > > > > > > > >
> > > > > > > > > > If clients also need a way to define it while
> interacting with
> > > > > > > Polaris
> > > > > > > > > over
> > > > > > > > > > the IRC API, let's discuss how that can be done.
> > > > > > > > > >
> > > > > > > > > > WDYT?
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Srinivas,
> > > > > > > > > > >
> > > > > > > > > > > I see  that PR 4023 is active again. I'll try and post
> a
> > > > fresh
> > > > > > > review
> > > > > > > > > > soon.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > > > > > > > > > sririshindra@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi JB and Dmitri,
> > > > > > > > > > >>
> > > > > > > > > > >> I raised PR #4023 <
> > > > > https://github.com/apache/polaris/pull/4023>
> > > > > > > > > > yesterday
> > > > > > > > > > >> as part of the broader per-table storage config
> effort, and
> > > > I
> > > > > > > > believe
> > > > > > > > > it
> > > > > > > > > > >> implements exactly what you are proposing.
> > > > > > > > > > >>
> > > > > > > > > > >> Building on the named storage configs introduced in
> PR #3409
> > > > > > > > > > >> <https://github.com/apache/polaris/pull/3409>, this
> PR
> > > > allows
> > > > > > > > > > namespaces
> > > > > > > > > > >> and tables to specify a polaris.storage.name
> property. This
> > > > > > acts
> > > > > > > as
> > > > > > > > > the
> > > > > > > > > > >> "identifier" JB mentioned, allowing credential
> vending to
> > > > > > resolve
> > > > > > > > the
> > > > > > > > > > >> correct, named storage config at runtime without
> introducing
> > > > > new
> > > > > > > > > > >> management
> > > > > > > > > > >> APIs.
> > > > > > > > > > >>
> > > > > > > > > > >> Could you please take a look at the PR and let me
> know if
> > > > this
> > > > > > > > aligns
> > > > > > > > > > with
> > > > > > > > > > >> what you are looking for?
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Yufei Gu <fl...@gmail.com>]

Hi Srinivas, Thanks for putting together the design doc and moving
this forward. The design doc looks good to me overall. Looking forward
to the PRs!

Yufei

On Fri, Aug 28, 2026 at 9:35 AM Alexandre Dutra <ad...@apache.org> wrote:
>
> Hi Srinivas, hi all,
>
> As Dmitri said, summer time indeed has slowed down the community
> bandwidth, but I'm happy to get back to this topic today.
>
> Thanks for the thorough writeup and also the summary of the current
> status quo: very useful. I'm OK with both the two-phase framing and
> the fail-closed approach.
>
> I have however one concern on Phase 1 that I think is worth settling
> before we move forward: the storageName field is being **overloaded to
> do two independent jobs**.
>
> Today (per #3409) this field has exactly one purpose: select the
> deployment-time credential set. Phase 1 quietly adds a second: it
> becomes the identity key of the named-config map and the target of
> polaris.storage.name. Those are two different namespaces: a
> catalog-scoped config identity vs. a deployment-scoped credential
> selector; welding them together means you can't share one credential
> set across configs, and you can't rename a config without re-pointing
> its credentials.
>
> This surfaces as two concrete rough edges:
>
> 1. The default config's storageName is ambiguous. §5.1 excludes the
> default's name from the named map, but §5.4 resolves references only
> against that map. So if the default carries storageName = "foo" and a
> table sets polaris.storage.name = "foo", the lookup misses and we fail
> closed with a "dangling reference", even though a config named "foo"
> plainly exists. **The default config having a name at all is a code
> smell.**
>
> 2. The two usages can disagree. If named references are found in the
> hierarchy, but RESOLVE_CREDENTIALS_BY_STORAGE_NAME is off, we'd
> resolve a table to a named storage config, but wouldn't pick named
> credentials for it, falling back to the default provider chain. That's
> a silent fail-open path inside a design tailored for fail-closed.
>
> My proposal: decouple the two concepts, and use a real map of name ->
> StorageConfigInfo in Create/UpdateCatalogRequest; credential selection
> keeps using the current storageName field (we could maybe rename it).
> The map key becomes the reference identity; uniqueness comes for free.
> That fully decouples the two concerns: several configs can share one
> credential set, and the default no longer needs a name.
>
> WDYT?
>
> Thanks,
> Alex
>
> On Mon, Aug 10, 2026 at 7:04 PM Dmitri Bourlatchkov <di...@apache.org> wrote:
> >
> > Hi Srinivas,
> >
> > Thanks for putting this document together. I think it provides an excellent
> > summary of this discussion and lays out clean path forward.
> >
> > As for phase 1 it looks like we could start implementing it, if other
> > reviews agree. Given that it's summer time it may take a week or two for
> > everybody to catch up on this thread, though.
> >
> > From a practical point of view, I tend to think that non-default storage
> > config can be use effectively only at the namespace level. For a new table
> > to be created via the IRC API, the request can only operate on the
> > name/config stored in the immediate namespace or higher. The table itself
> > cannot define the storage name unless we revisit the point of managing it
> > only via Polaris properties.
> >
> > It might be worth allowing Iceberg table properties to carry the storage
> > name too. Polaris can forward that information to its internal properties
> > on changes. However, this point does not affect Phase 1, I think. We can
> > add this kind of property forwarding later.
> >
> > Cheers,
> > Dmitri.
> >
> > On Wed, Aug 5, 2026 at 10:51 AM Srinivas Rishindra <sr...@gmail.com>
> > wrote:
> >
> > > I've put together an updated design proposal document detailing the storage
> > > configuration management proposal. Please provide your feedback on the
> > > proposal.
> > >
> > >
> > > https://docs.google.com/document/d/1TSaZ0G2oEoWq6lDMSPIIfHLwXxW9VZmhUNE3s_jfxgI/edit?usp=sharing
> > >
> > >
> > >
> > > On Thu, Jun 18, 2026 at 3:23 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Adam,
> > > >
> > > > Could you give some more details, please, about where you think the
> > > Iceberg
> > > > spec disallows multiple storage per table?
> > > >
> > > > I never thought it was not allowed. Perhaps it was inconvenient or
> > > > impractical in existing clients, but the spec itself seems to be fairly
> > > > unbiased with respect to actual storage technology... Perhaps I missed
> > > > something, though.
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <sz...@apache.org> wrote:
> > > >
> > > > > Hi Yufei,
> > > > >
> > > > > Regarding #2:
> > > > > Would opting for a simple (assuming per table) storageConfigName or
> > > > > storageConfigId
> > > > > support future extensibility?
> > > > > Asking from a multiple-storage-per-table perspective as it's something
> > > > that
> > > > > the Iceberg spec does allow IMHO.
> > > > >
> > > > > Cheers,
> > > > > Adam
> > > > >
> > > > > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <fl...@gmail.com> wrote:
> > > > >
> > > > > > Hi folks, thanks for the discussion. They are really helpful!
> > > > > >
> > > > > > I think there are two separate questions here:
> > > > > >
> > > > > > 1. How should we store storage configurations? As catalog properties
> > > or
> > > > > as
> > > > > > separate entities?
> > > > > > I'd lean toward separate entities in the long term. It provides a
> > > > cleaner
> > > > > > boundary between a catalog and its storage configurations, avoids
> > > > > bloating
> > > > > > the catalog object as the number of configs grows, and gives us a
> > > > better
> > > > > > foundation for future access control since entity level permissions
> > > are
> > > > > > easier to reason about.
> > > > > >
> > > > > > 2. How should a table or namespace refer to a storage configuration?
> > > > > > I'd prefer storing a simple reference on the entity itself rather
> > > than
> > > > > > introducing a separate relationship table. Both approaches support
> > > > > > hierarchical lookup and inheritance when a storage config is not
> > > > > specified
> > > > > > at the current level. However, I don't think we should allow multiple
> > > > > > storage configs to be associated with a single table or namespace.
> > > > Given
> > > > > > that cardinality, a simple storageConfigName or storageConfigId field
> > > > > seems
> > > > > > sufficient and easier to understand than a dedicated relationship
> > > > model.
> > > > > >
> > > > > > For example:
> > > > > >
> > > > > > Catalog
> > > > > > ├─ entity: StorageConfigA
> > > > > > ├─ entity: StorageConfigB
> > > > > > └─ entity: StorageConfigC
> > > > > >
> > > > > > Table1 storage property -> StorageConfigA
> > > > > > Table2 storage property -> StorageConfigB
> > > > > > Table3 storage property -> StorageConfigC
> > > > > >
> > > > > > To me, this keeps the model simple while still supporting shared
> > > > storage
> > > > > > configurations and future extensibility.
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Robert,
> > > > > > >
> > > > > > > The distinction between "attached" and "referenced" config makes
> > > > sense.
> > > > > > >
> > > > > > > My impression from JB's message on Mar 19 in this thread [1] is
> > > that
> > > > JB
> > > > > > is
> > > > > > > specifically proposing the "referenced" model, though.
> > > > > > >
> > > > > > > From my POV, both approaches can work, but the "referenced" model
> > > > might
> > > > > > be
> > > > > > > more intuitive to users.
> > > > > > >
> > > > > > > [1]
> > > https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <sn...@snazy.de>
> > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > AFAIK the current Polaris code appears to resolve the storage
> > > > > > > > configuration from the leaf (table), upwards via the namespace
> > > > > elements
> > > > > > > to
> > > > > > > > the catalog, choosing the first one found.
> > > > > > > > I am not sure whether that functionality is already used
> > > anywhere?
> > > > > > > > Being able to use a storage configuration on a namespace or a
> > > leaf
> > > > > > > appears
> > > > > > > > appealing.
> > > > > > > >
> > > > > > > > OTOH I see a demand for named storage configurations.
> > > > > > > >
> > > > > > > > I wonder whether the mechanism to derive the storage
> > > configuration
> > > > > > could
> > > > > > > > support both "directly attached" and "referenced" storage
> > > > > > configurations:
> > > > > > > > A namespace element or leaf can have either a "directly attached"
> > > > > > storage
> > > > > > > > configuration or a reference to a named one.
> > > > > > > >
> > > > > > > > We'd need management-ish APIs to handle the storage
> > > configurations
> > > > > for
> > > > > > a
> > > > > > > > namespace or leaf.
> > > > > > > > These APIs, which do not exist yet, could start with "directly
> > > > > > attached"
> > > > > > > > storage configurations and once the authz questions are resolved,
> > > > > > extend
> > > > > > > to
> > > > > > > > include named, shared storage configurations.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > Robert
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > To recap the discussion from the Community Sync today: My
> > > > > > understanding
> > > > > > > > is
> > > > > > > > > that the consensus was:
> > > > > > > > >
> > > > > > > > > * Defer AuthZ aspects for later discussion.
> > > > > > > > >
> > > > > > > > > * Allow multiple Storage Config objects per Catalog. If more
> > > than
> > > > > one
> > > > > > > > > Storage Config is present, they are to have different (within
> > > the
> > > > > > > > catalog)
> > > > > > > > > names. Rishi to open a new PR for this (scoped to REST API +
> > > > Sever
> > > > > > > > > changes).
> > > > > > > > >
> > > > > > > > > * Defer CLI changes until we have the full end-to-end system
> > > > > working
> > > > > > > (it
> > > > > > > > > should be usable via curl)
> > > > > > > > >
> > > > > > > > > * PR 4023 is on hold for now. It is to be rebased after adding
> > > > > > support
> > > > > > > > for
> > > > > > > > > multiple Storage Configs
> > > > > > > > >
> > > > > > > > > * The connection between storage names and storage access
> > > > > credentials
> > > > > > > is
> > > > > > > > to
> > > > > > > > > remain unchanged for now. It is effectively a deployment time
> > > > > concern
> > > > > > > > given
> > > > > > > > > the current OSS code.
> > > > > > > > >
> > > > > > > > > Please correct / add if I missed anything.
> > > > > > > > >
> > > > > > > > > From my POV, I think we still need to discuss how exactly
> > > > entities
> > > > > > > > (tables,
> > > > > > > > > views) will link to Storage Config before we can resume PR
> > > 4023.
> > > > > > > > >
> > > > > > > > > I propose storing the config name in the Entity properties (not
> > > > in
> > > > > > > Table
> > > > > > > > > Metadata properties).
> > > > > > > > >
> > > > > > > > > If clients also need a way to define it while interacting with
> > > > > > Polaris
> > > > > > > > over
> > > > > > > > > the IRC API, let's discuss how that can be done.
> > > > > > > > >
> > > > > > > > > WDYT?
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Srinivas,
> > > > > > > > > >
> > > > > > > > > > I see  that PR 4023 is active again. I'll try and post a
> > > fresh
> > > > > > review
> > > > > > > > > soon.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > > > > > > > > sririshindra@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi JB and Dmitri,
> > > > > > > > > >>
> > > > > > > > > >> I raised PR #4023 <
> > > > https://github.com/apache/polaris/pull/4023>
> > > > > > > > > yesterday
> > > > > > > > > >> as part of the broader per-table storage config effort, and
> > > I
> > > > > > > believe
> > > > > > > > it
> > > > > > > > > >> implements exactly what you are proposing.
> > > > > > > > > >>
> > > > > > > > > >> Building on the named storage configs introduced in PR #3409
> > > > > > > > > >> <https://github.com/apache/polaris/pull/3409>, this PR
> > > allows
> > > > > > > > > namespaces
> > > > > > > > > >> and tables to specify a polaris.storage.name property. This
> > > > > acts
> > > > > > as
> > > > > > > > the
> > > > > > > > > >> "identifier" JB mentioned, allowing credential vending to
> > > > > resolve
> > > > > > > the
> > > > > > > > > >> correct, named storage config at runtime without introducing
> > > > new
> > > > > > > > > >> management
> > > > > > > > > >> APIs.
> > > > > > > > > >>
> > > > > > > > > >> Could you please take a look at the PR and let me know if
> > > this
> > > > > > > aligns
> > > > > > > > > with
> > > > > > > > > >> what you are looking for?
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Alexandre Dutra <ad...@apache.org>]

Hi Srinivas, hi all,

As Dmitri said, summer time indeed has slowed down the community
bandwidth, but I'm happy to get back to this topic today.

Thanks for the thorough writeup and also the summary of the current
status quo: very useful. I'm OK with both the two-phase framing and
the fail-closed approach.

I have however one concern on Phase 1 that I think is worth settling
before we move forward: the storageName field is being **overloaded to
do two independent jobs**.

Today (per #3409) this field has exactly one purpose: select the
deployment-time credential set. Phase 1 quietly adds a second: it
becomes the identity key of the named-config map and the target of
polaris.storage.name. Those are two different namespaces: a
catalog-scoped config identity vs. a deployment-scoped credential
selector; welding them together means you can't share one credential
set across configs, and you can't rename a config without re-pointing
its credentials.

This surfaces as two concrete rough edges:

1. The default config's storageName is ambiguous. §5.1 excludes the
default's name from the named map, but §5.4 resolves references only
against that map. So if the default carries storageName = "foo" and a
table sets polaris.storage.name = "foo", the lookup misses and we fail
closed with a "dangling reference", even though a config named "foo"
plainly exists. **The default config having a name at all is a code
smell.**

2. The two usages can disagree. If named references are found in the
hierarchy, but RESOLVE_CREDENTIALS_BY_STORAGE_NAME is off, we'd
resolve a table to a named storage config, but wouldn't pick named
credentials for it, falling back to the default provider chain. That's
a silent fail-open path inside a design tailored for fail-closed.

My proposal: decouple the two concepts, and use a real map of name ->
StorageConfigInfo in Create/UpdateCatalogRequest; credential selection
keeps using the current storageName field (we could maybe rename it).
The map key becomes the reference identity; uniqueness comes for free.
That fully decouples the two concerns: several configs can share one
credential set, and the default no longer needs a name.

WDYT?

Thanks,
Alex

On Mon, Aug 10, 2026 at 7:04 PM Dmitri Bourlatchkov <di...@apache.org> wrote:
>
> Hi Srinivas,
>
> Thanks for putting this document together. I think it provides an excellent
> summary of this discussion and lays out clean path forward.
>
> As for phase 1 it looks like we could start implementing it, if other
> reviews agree. Given that it's summer time it may take a week or two for
> everybody to catch up on this thread, though.
>
> From a practical point of view, I tend to think that non-default storage
> config can be use effectively only at the namespace level. For a new table
> to be created via the IRC API, the request can only operate on the
> name/config stored in the immediate namespace or higher. The table itself
> cannot define the storage name unless we revisit the point of managing it
> only via Polaris properties.
>
> It might be worth allowing Iceberg table properties to carry the storage
> name too. Polaris can forward that information to its internal properties
> on changes. However, this point does not affect Phase 1, I think. We can
> add this kind of property forwarding later.
>
> Cheers,
> Dmitri.
>
> On Wed, Aug 5, 2026 at 10:51 AM Srinivas Rishindra <sr...@gmail.com>
> wrote:
>
> > I've put together an updated design proposal document detailing the storage
> > configuration management proposal. Please provide your feedback on the
> > proposal.
> >
> >
> > https://docs.google.com/document/d/1TSaZ0G2oEoWq6lDMSPIIfHLwXxW9VZmhUNE3s_jfxgI/edit?usp=sharing
> >
> >
> >
> > On Thu, Jun 18, 2026 at 3:23 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Adam,
> > >
> > > Could you give some more details, please, about where you think the
> > Iceberg
> > > spec disallows multiple storage per table?
> > >
> > > I never thought it was not allowed. Perhaps it was inconvenient or
> > > impractical in existing clients, but the spec itself seems to be fairly
> > > unbiased with respect to actual storage technology... Perhaps I missed
> > > something, though.
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <sz...@apache.org> wrote:
> > >
> > > > Hi Yufei,
> > > >
> > > > Regarding #2:
> > > > Would opting for a simple (assuming per table) storageConfigName or
> > > > storageConfigId
> > > > support future extensibility?
> > > > Asking from a multiple-storage-per-table perspective as it's something
> > > that
> > > > the Iceberg spec does allow IMHO.
> > > >
> > > > Cheers,
> > > > Adam
> > > >
> > > > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <fl...@gmail.com> wrote:
> > > >
> > > > > Hi folks, thanks for the discussion. They are really helpful!
> > > > >
> > > > > I think there are two separate questions here:
> > > > >
> > > > > 1. How should we store storage configurations? As catalog properties
> > or
> > > > as
> > > > > separate entities?
> > > > > I'd lean toward separate entities in the long term. It provides a
> > > cleaner
> > > > > boundary between a catalog and its storage configurations, avoids
> > > > bloating
> > > > > the catalog object as the number of configs grows, and gives us a
> > > better
> > > > > foundation for future access control since entity level permissions
> > are
> > > > > easier to reason about.
> > > > >
> > > > > 2. How should a table or namespace refer to a storage configuration?
> > > > > I'd prefer storing a simple reference on the entity itself rather
> > than
> > > > > introducing a separate relationship table. Both approaches support
> > > > > hierarchical lookup and inheritance when a storage config is not
> > > > specified
> > > > > at the current level. However, I don't think we should allow multiple
> > > > > storage configs to be associated with a single table or namespace.
> > > Given
> > > > > that cardinality, a simple storageConfigName or storageConfigId field
> > > > seems
> > > > > sufficient and easier to understand than a dedicated relationship
> > > model.
> > > > >
> > > > > For example:
> > > > >
> > > > > Catalog
> > > > > ├─ entity: StorageConfigA
> > > > > ├─ entity: StorageConfigB
> > > > > └─ entity: StorageConfigC
> > > > >
> > > > > Table1 storage property -> StorageConfigA
> > > > > Table2 storage property -> StorageConfigB
> > > > > Table3 storage property -> StorageConfigC
> > > > >
> > > > > To me, this keeps the model simple while still supporting shared
> > > storage
> > > > > configurations and future extensibility.
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Robert,
> > > > > >
> > > > > > The distinction between "attached" and "referenced" config makes
> > > sense.
> > > > > >
> > > > > > My impression from JB's message on Mar 19 in this thread [1] is
> > that
> > > JB
> > > > > is
> > > > > > specifically proposing the "referenced" model, though.
> > > > > >
> > > > > > From my POV, both approaches can work, but the "referenced" model
> > > might
> > > > > be
> > > > > > more intuitive to users.
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <sn...@snazy.de>
> > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > AFAIK the current Polaris code appears to resolve the storage
> > > > > > > configuration from the leaf (table), upwards via the namespace
> > > > elements
> > > > > > to
> > > > > > > the catalog, choosing the first one found.
> > > > > > > I am not sure whether that functionality is already used
> > anywhere?
> > > > > > > Being able to use a storage configuration on a namespace or a
> > leaf
> > > > > > appears
> > > > > > > appealing.
> > > > > > >
> > > > > > > OTOH I see a demand for named storage configurations.
> > > > > > >
> > > > > > > I wonder whether the mechanism to derive the storage
> > configuration
> > > > > could
> > > > > > > support both "directly attached" and "referenced" storage
> > > > > configurations:
> > > > > > > A namespace element or leaf can have either a "directly attached"
> > > > > storage
> > > > > > > configuration or a reference to a named one.
> > > > > > >
> > > > > > > We'd need management-ish APIs to handle the storage
> > configurations
> > > > for
> > > > > a
> > > > > > > namespace or leaf.
> > > > > > > These APIs, which do not exist yet, could start with "directly
> > > > > attached"
> > > > > > > storage configurations and once the authz questions are resolved,
> > > > > extend
> > > > > > to
> > > > > > > include named, shared storage configurations.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Robert
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > To recap the discussion from the Community Sync today: My
> > > > > understanding
> > > > > > > is
> > > > > > > > that the consensus was:
> > > > > > > >
> > > > > > > > * Defer AuthZ aspects for later discussion.
> > > > > > > >
> > > > > > > > * Allow multiple Storage Config objects per Catalog. If more
> > than
> > > > one
> > > > > > > > Storage Config is present, they are to have different (within
> > the
> > > > > > > catalog)
> > > > > > > > names. Rishi to open a new PR for this (scoped to REST API +
> > > Sever
> > > > > > > > changes).
> > > > > > > >
> > > > > > > > * Defer CLI changes until we have the full end-to-end system
> > > > working
> > > > > > (it
> > > > > > > > should be usable via curl)
> > > > > > > >
> > > > > > > > * PR 4023 is on hold for now. It is to be rebased after adding
> > > > > support
> > > > > > > for
> > > > > > > > multiple Storage Configs
> > > > > > > >
> > > > > > > > * The connection between storage names and storage access
> > > > credentials
> > > > > > is
> > > > > > > to
> > > > > > > > remain unchanged for now. It is effectively a deployment time
> > > > concern
> > > > > > > given
> > > > > > > > the current OSS code.
> > > > > > > >
> > > > > > > > Please correct / add if I missed anything.
> > > > > > > >
> > > > > > > > From my POV, I think we still need to discuss how exactly
> > > entities
> > > > > > > (tables,
> > > > > > > > views) will link to Storage Config before we can resume PR
> > 4023.
> > > > > > > >
> > > > > > > > I propose storing the config name in the Entity properties (not
> > > in
> > > > > > Table
> > > > > > > > Metadata properties).
> > > > > > > >
> > > > > > > > If clients also need a way to define it while interacting with
> > > > > Polaris
> > > > > > > over
> > > > > > > > the IRC API, let's discuss how that can be done.
> > > > > > > >
> > > > > > > > WDYT?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Srinivas,
> > > > > > > > >
> > > > > > > > > I see  that PR 4023 is active again. I'll try and post a
> > fresh
> > > > > review
> > > > > > > > soon.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > > > > > > > sririshindra@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi JB and Dmitri,
> > > > > > > > >>
> > > > > > > > >> I raised PR #4023 <
> > > https://github.com/apache/polaris/pull/4023>
> > > > > > > > yesterday
> > > > > > > > >> as part of the broader per-table storage config effort, and
> > I
> > > > > > believe
> > > > > > > it
> > > > > > > > >> implements exactly what you are proposing.
> > > > > > > > >>
> > > > > > > > >> Building on the named storage configs introduced in PR #3409
> > > > > > > > >> <https://github.com/apache/polaris/pull/3409>, this PR
> > allows
> > > > > > > > namespaces
> > > > > > > > >> and tables to specify a polaris.storage.name property. This
> > > > acts
> > > > > as
> > > > > > > the
> > > > > > > > >> "identifier" JB mentioned, allowing credential vending to
> > > > resolve
> > > > > > the
> > > > > > > > >> correct, named storage config at runtime without introducing
> > > new
> > > > > > > > >> management
> > > > > > > > >> APIs.
> > > > > > > > >>
> > > > > > > > >> Could you please take a look at the PR and let me know if
> > this
> > > > > > aligns
> > > > > > > > with
> > > > > > > > >> what you are looking for?
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Srinivas,

Thanks for putting this document together. I think it provides an excellent
summary of this discussion and lays out clean path forward.

As for phase 1 it looks like we could start implementing it, if other
reviews agree. Given that it's summer time it may take a week or two for
everybody to catch up on this thread, though.

From a practical point of view, I tend to think that non-default storage
config can be use effectively only at the namespace level. For a new table
to be created via the IRC API, the request can only operate on the
name/config stored in the immediate namespace or higher. The table itself
cannot define the storage name unless we revisit the point of managing it
only via Polaris properties.

It might be worth allowing Iceberg table properties to carry the storage
name too. Polaris can forward that information to its internal properties
on changes. However, this point does not affect Phase 1, I think. We can
add this kind of property forwarding later.

Cheers,
Dmitri.

On Wed, Aug 5, 2026 at 10:51 AM Srinivas Rishindra <sr...@gmail.com>
wrote:

> I've put together an updated design proposal document detailing the storage
> configuration management proposal. Please provide your feedback on the
> proposal.
>
>
> https://docs.google.com/document/d/1TSaZ0G2oEoWq6lDMSPIIfHLwXxW9VZmhUNE3s_jfxgI/edit?usp=sharing
>
>
>
> On Thu, Jun 18, 2026 at 3:23 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Adam,
> >
> > Could you give some more details, please, about where you think the
> Iceberg
> > spec disallows multiple storage per table?
> >
> > I never thought it was not allowed. Perhaps it was inconvenient or
> > impractical in existing clients, but the spec itself seems to be fairly
> > unbiased with respect to actual storage technology... Perhaps I missed
> > something, though.
> >
> > Thanks,
> > Dmitri.
> >
> > On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <sz...@apache.org> wrote:
> >
> > > Hi Yufei,
> > >
> > > Regarding #2:
> > > Would opting for a simple (assuming per table) storageConfigName or
> > > storageConfigId
> > > support future extensibility?
> > > Asking from a multiple-storage-per-table perspective as it's something
> > that
> > > the Iceberg spec does allow IMHO.
> > >
> > > Cheers,
> > > Adam
> > >
> > > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > > Hi folks, thanks for the discussion. They are really helpful!
> > > >
> > > > I think there are two separate questions here:
> > > >
> > > > 1. How should we store storage configurations? As catalog properties
> or
> > > as
> > > > separate entities?
> > > > I'd lean toward separate entities in the long term. It provides a
> > cleaner
> > > > boundary between a catalog and its storage configurations, avoids
> > > bloating
> > > > the catalog object as the number of configs grows, and gives us a
> > better
> > > > foundation for future access control since entity level permissions
> are
> > > > easier to reason about.
> > > >
> > > > 2. How should a table or namespace refer to a storage configuration?
> > > > I'd prefer storing a simple reference on the entity itself rather
> than
> > > > introducing a separate relationship table. Both approaches support
> > > > hierarchical lookup and inheritance when a storage config is not
> > > specified
> > > > at the current level. However, I don't think we should allow multiple
> > > > storage configs to be associated with a single table or namespace.
> > Given
> > > > that cardinality, a simple storageConfigName or storageConfigId field
> > > seems
> > > > sufficient and easier to understand than a dedicated relationship
> > model.
> > > >
> > > > For example:
> > > >
> > > > Catalog
> > > > ├─ entity: StorageConfigA
> > > > ├─ entity: StorageConfigB
> > > > └─ entity: StorageConfigC
> > > >
> > > > Table1 storage property -> StorageConfigA
> > > > Table2 storage property -> StorageConfigB
> > > > Table3 storage property -> StorageConfigC
> > > >
> > > > To me, this keeps the model simple while still supporting shared
> > storage
> > > > configurations and future extensibility.
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Robert,
> > > > >
> > > > > The distinction between "attached" and "referenced" config makes
> > sense.
> > > > >
> > > > > My impression from JB's message on Mar 19 in this thread [1] is
> that
> > JB
> > > > is
> > > > > specifically proposing the "referenced" model, though.
> > > > >
> > > > > From my POV, both approaches can work, but the "referenced" model
> > might
> > > > be
> > > > > more intuitive to users.
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <sn...@snazy.de>
> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > AFAIK the current Polaris code appears to resolve the storage
> > > > > > configuration from the leaf (table), upwards via the namespace
> > > elements
> > > > > to
> > > > > > the catalog, choosing the first one found.
> > > > > > I am not sure whether that functionality is already used
> anywhere?
> > > > > > Being able to use a storage configuration on a namespace or a
> leaf
> > > > > appears
> > > > > > appealing.
> > > > > >
> > > > > > OTOH I see a demand for named storage configurations.
> > > > > >
> > > > > > I wonder whether the mechanism to derive the storage
> configuration
> > > > could
> > > > > > support both "directly attached" and "referenced" storage
> > > > configurations:
> > > > > > A namespace element or leaf can have either a "directly attached"
> > > > storage
> > > > > > configuration or a reference to a named one.
> > > > > >
> > > > > > We'd need management-ish APIs to handle the storage
> configurations
> > > for
> > > > a
> > > > > > namespace or leaf.
> > > > > > These APIs, which do not exist yet, could start with "directly
> > > > attached"
> > > > > > storage configurations and once the authz questions are resolved,
> > > > extend
> > > > > to
> > > > > > include named, shared storage configurations.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Robert
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > To recap the discussion from the Community Sync today: My
> > > > understanding
> > > > > > is
> > > > > > > that the consensus was:
> > > > > > >
> > > > > > > * Defer AuthZ aspects for later discussion.
> > > > > > >
> > > > > > > * Allow multiple Storage Config objects per Catalog. If more
> than
> > > one
> > > > > > > Storage Config is present, they are to have different (within
> the
> > > > > > catalog)
> > > > > > > names. Rishi to open a new PR for this (scoped to REST API +
> > Sever
> > > > > > > changes).
> > > > > > >
> > > > > > > * Defer CLI changes until we have the full end-to-end system
> > > working
> > > > > (it
> > > > > > > should be usable via curl)
> > > > > > >
> > > > > > > * PR 4023 is on hold for now. It is to be rebased after adding
> > > > support
> > > > > > for
> > > > > > > multiple Storage Configs
> > > > > > >
> > > > > > > * The connection between storage names and storage access
> > > credentials
> > > > > is
> > > > > > to
> > > > > > > remain unchanged for now. It is effectively a deployment time
> > > concern
> > > > > > given
> > > > > > > the current OSS code.
> > > > > > >
> > > > > > > Please correct / add if I missed anything.
> > > > > > >
> > > > > > > From my POV, I think we still need to discuss how exactly
> > entities
> > > > > > (tables,
> > > > > > > views) will link to Storage Config before we can resume PR
> 4023.
> > > > > > >
> > > > > > > I propose storing the config name in the Entity properties (not
> > in
> > > > > Table
> > > > > > > Metadata properties).
> > > > > > >
> > > > > > > If clients also need a way to define it while interacting with
> > > > Polaris
> > > > > > over
> > > > > > > the IRC API, let's discuss how that can be done.
> > > > > > >
> > > > > > > WDYT?
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <
> > > dimas@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Srinivas,
> > > > > > > >
> > > > > > > > I see  that PR 4023 is active again. I'll try and post a
> fresh
> > > > review
> > > > > > > soon.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > > > > > > sririshindra@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi JB and Dmitri,
> > > > > > > >>
> > > > > > > >> I raised PR #4023 <
> > https://github.com/apache/polaris/pull/4023>
> > > > > > > yesterday
> > > > > > > >> as part of the broader per-table storage config effort, and
> I
> > > > > believe
> > > > > > it
> > > > > > > >> implements exactly what you are proposing.
> > > > > > > >>
> > > > > > > >> Building on the named storage configs introduced in PR #3409
> > > > > > > >> <https://github.com/apache/polaris/pull/3409>, this PR
> allows
> > > > > > > namespaces
> > > > > > > >> and tables to specify a polaris.storage.name property. This
> > > acts
> > > > as
> > > > > > the
> > > > > > > >> "identifier" JB mentioned, allowing credential vending to
> > > resolve
> > > > > the
> > > > > > > >> correct, named storage config at runtime without introducing
> > new
> > > > > > > >> management
> > > > > > > >> APIs.
> > > > > > > >>
> > > > > > > >> Could you please take a look at the PR and let me know if
> this
> > > > > aligns
> > > > > > > with
> > > > > > > >> what you are looking for?
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Srinivas Rishindra <sr...@gmail.com>]

I've put together an updated design proposal document detailing the storage
configuration management proposal. Please provide your feedback on the
proposal.

https://docs.google.com/document/d/1TSaZ0G2oEoWq6lDMSPIIfHLwXxW9VZmhUNE3s_jfxgI/edit?usp=sharing



On Thu, Jun 18, 2026 at 3:23 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi Adam,
>
> Could you give some more details, please, about where you think the Iceberg
> spec disallows multiple storage per table?
>
> I never thought it was not allowed. Perhaps it was inconvenient or
> impractical in existing clients, but the spec itself seems to be fairly
> unbiased with respect to actual storage technology... Perhaps I missed
> something, though.
>
> Thanks,
> Dmitri.
>
> On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <sz...@apache.org> wrote:
>
> > Hi Yufei,
> >
> > Regarding #2:
> > Would opting for a simple (assuming per table) storageConfigName or
> > storageConfigId
> > support future extensibility?
> > Asking from a multiple-storage-per-table perspective as it's something
> that
> > the Iceberg spec does allow IMHO.
> >
> > Cheers,
> > Adam
> >
> > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Hi folks, thanks for the discussion. They are really helpful!
> > >
> > > I think there are two separate questions here:
> > >
> > > 1. How should we store storage configurations? As catalog properties or
> > as
> > > separate entities?
> > > I'd lean toward separate entities in the long term. It provides a
> cleaner
> > > boundary between a catalog and its storage configurations, avoids
> > bloating
> > > the catalog object as the number of configs grows, and gives us a
> better
> > > foundation for future access control since entity level permissions are
> > > easier to reason about.
> > >
> > > 2. How should a table or namespace refer to a storage configuration?
> > > I'd prefer storing a simple reference on the entity itself rather than
> > > introducing a separate relationship table. Both approaches support
> > > hierarchical lookup and inheritance when a storage config is not
> > specified
> > > at the current level. However, I don't think we should allow multiple
> > > storage configs to be associated with a single table or namespace.
> Given
> > > that cardinality, a simple storageConfigName or storageConfigId field
> > seems
> > > sufficient and easier to understand than a dedicated relationship
> model.
> > >
> > > For example:
> > >
> > > Catalog
> > > ├─ entity: StorageConfigA
> > > ├─ entity: StorageConfigB
> > > └─ entity: StorageConfigC
> > >
> > > Table1 storage property -> StorageConfigA
> > > Table2 storage property -> StorageConfigB
> > > Table3 storage property -> StorageConfigC
> > >
> > > To me, this keeps the model simple while still supporting shared
> storage
> > > configurations and future extensibility.
> > >
> > > Yufei
> > >
> > >
> > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Robert,
> > > >
> > > > The distinction between "attached" and "referenced" config makes
> sense.
> > > >
> > > > My impression from JB's message on Mar 19 in this thread [1] is that
> JB
> > > is
> > > > specifically proposing the "referenced" model, though.
> > > >
> > > > From my POV, both approaches can work, but the "referenced" model
> might
> > > be
> > > > more intuitive to users.
> > > >
> > > > [1] https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <sn...@snazy.de> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > AFAIK the current Polaris code appears to resolve the storage
> > > > > configuration from the leaf (table), upwards via the namespace
> > elements
> > > > to
> > > > > the catalog, choosing the first one found.
> > > > > I am not sure whether that functionality is already used anywhere?
> > > > > Being able to use a storage configuration on a namespace or a leaf
> > > > appears
> > > > > appealing.
> > > > >
> > > > > OTOH I see a demand for named storage configurations.
> > > > >
> > > > > I wonder whether the mechanism to derive the storage configuration
> > > could
> > > > > support both "directly attached" and "referenced" storage
> > > configurations:
> > > > > A namespace element or leaf can have either a "directly attached"
> > > storage
> > > > > configuration or a reference to a named one.
> > > > >
> > > > > We'd need management-ish APIs to handle the storage configurations
> > for
> > > a
> > > > > namespace or leaf.
> > > > > These APIs, which do not exist yet, could start with "directly
> > > attached"
> > > > > storage configurations and once the authz questions are resolved,
> > > extend
> > > > to
> > > > > include named, shared storage configurations.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Robert
> > > > >
> > > > >
> > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > To recap the discussion from the Community Sync today: My
> > > understanding
> > > > > is
> > > > > > that the consensus was:
> > > > > >
> > > > > > * Defer AuthZ aspects for later discussion.
> > > > > >
> > > > > > * Allow multiple Storage Config objects per Catalog. If more than
> > one
> > > > > > Storage Config is present, they are to have different (within the
> > > > > catalog)
> > > > > > names. Rishi to open a new PR for this (scoped to REST API +
> Sever
> > > > > > changes).
> > > > > >
> > > > > > * Defer CLI changes until we have the full end-to-end system
> > working
> > > > (it
> > > > > > should be usable via curl)
> > > > > >
> > > > > > * PR 4023 is on hold for now. It is to be rebased after adding
> > > support
> > > > > for
> > > > > > multiple Storage Configs
> > > > > >
> > > > > > * The connection between storage names and storage access
> > credentials
> > > > is
> > > > > to
> > > > > > remain unchanged for now. It is effectively a deployment time
> > concern
> > > > > given
> > > > > > the current OSS code.
> > > > > >
> > > > > > Please correct / add if I missed anything.
> > > > > >
> > > > > > From my POV, I think we still need to discuss how exactly
> entities
> > > > > (tables,
> > > > > > views) will link to Storage Config before we can resume PR 4023.
> > > > > >
> > > > > > I propose storing the config name in the Entity properties (not
> in
> > > > Table
> > > > > > Metadata properties).
> > > > > >
> > > > > > If clients also need a way to define it while interacting with
> > > Polaris
> > > > > over
> > > > > > the IRC API, let's discuss how that can be done.
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <
> > dimas@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Srinivas,
> > > > > > >
> > > > > > > I see  that PR 4023 is active again. I'll try and post a fresh
> > > review
> > > > > > soon.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > > > > > sririshindra@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi JB and Dmitri,
> > > > > > >>
> > > > > > >> I raised PR #4023 <
> https://github.com/apache/polaris/pull/4023>
> > > > > > yesterday
> > > > > > >> as part of the broader per-table storage config effort, and I
> > > > believe
> > > > > it
> > > > > > >> implements exactly what you are proposing.
> > > > > > >>
> > > > > > >> Building on the named storage configs introduced in PR #3409
> > > > > > >> <https://github.com/apache/polaris/pull/3409>, this PR allows
> > > > > > namespaces
> > > > > > >> and tables to specify a polaris.storage.name property. This
> > acts
> > > as
> > > > > the
> > > > > > >> "identifier" JB mentioned, allowing credential vending to
> > resolve
> > > > the
> > > > > > >> correct, named storage config at runtime without introducing
> new
> > > > > > >> management
> > > > > > >> APIs.
> > > > > > >>
> > > > > > >> Could you please take a look at the PR and let me know if this
> > > > aligns
> > > > > > with
> > > > > > >> what you are looking for?
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Adam,

Could you give some more details, please, about where you think the Iceberg
spec disallows multiple storage per table?

I never thought it was not allowed. Perhaps it was inconvenient or
impractical in existing clients, but the spec itself seems to be fairly
unbiased with respect to actual storage technology... Perhaps I missed
something, though.

Thanks,
Dmitri.

On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <sz...@apache.org> wrote:

> Hi Yufei,
>
> Regarding #2:
> Would opting for a simple (assuming per table) storageConfigName or
> storageConfigId
> support future extensibility?
> Asking from a multiple-storage-per-table perspective as it's something that
> the Iceberg spec does allow IMHO.
>
> Cheers,
> Adam
>
> On Wed, 17 Jun 2026 at 00:37, Yufei Gu <fl...@gmail.com> wrote:
>
> > Hi folks, thanks for the discussion. They are really helpful!
> >
> > I think there are two separate questions here:
> >
> > 1. How should we store storage configurations? As catalog properties or
> as
> > separate entities?
> > I'd lean toward separate entities in the long term. It provides a cleaner
> > boundary between a catalog and its storage configurations, avoids
> bloating
> > the catalog object as the number of configs grows, and gives us a better
> > foundation for future access control since entity level permissions are
> > easier to reason about.
> >
> > 2. How should a table or namespace refer to a storage configuration?
> > I'd prefer storing a simple reference on the entity itself rather than
> > introducing a separate relationship table. Both approaches support
> > hierarchical lookup and inheritance when a storage config is not
> specified
> > at the current level. However, I don't think we should allow multiple
> > storage configs to be associated with a single table or namespace. Given
> > that cardinality, a simple storageConfigName or storageConfigId field
> seems
> > sufficient and easier to understand than a dedicated relationship model.
> >
> > For example:
> >
> > Catalog
> > ├─ entity: StorageConfigA
> > ├─ entity: StorageConfigB
> > └─ entity: StorageConfigC
> >
> > Table1 storage property -> StorageConfigA
> > Table2 storage property -> StorageConfigB
> > Table3 storage property -> StorageConfigC
> >
> > To me, this keeps the model simple while still supporting shared storage
> > configurations and future extensibility.
> >
> > Yufei
> >
> >
> > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Robert,
> > >
> > > The distinction between "attached" and "referenced" config makes sense.
> > >
> > > My impression from JB's message on Mar 19 in this thread [1] is that JB
> > is
> > > specifically proposing the "referenced" model, though.
> > >
> > > From my POV, both approaches can work, but the "referenced" model might
> > be
> > > more intuitive to users.
> > >
> > > [1] https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <sn...@snazy.de> wrote:
> > >
> > > > Hi all,
> > > >
> > > > AFAIK the current Polaris code appears to resolve the storage
> > > > configuration from the leaf (table), upwards via the namespace
> elements
> > > to
> > > > the catalog, choosing the first one found.
> > > > I am not sure whether that functionality is already used anywhere?
> > > > Being able to use a storage configuration on a namespace or a leaf
> > > appears
> > > > appealing.
> > > >
> > > > OTOH I see a demand for named storage configurations.
> > > >
> > > > I wonder whether the mechanism to derive the storage configuration
> > could
> > > > support both "directly attached" and "referenced" storage
> > configurations:
> > > > A namespace element or leaf can have either a "directly attached"
> > storage
> > > > configuration or a reference to a named one.
> > > >
> > > > We'd need management-ish APIs to handle the storage configurations
> for
> > a
> > > > namespace or leaf.
> > > > These APIs, which do not exist yet, could start with "directly
> > attached"
> > > > storage configurations and once the authz questions are resolved,
> > extend
> > > to
> > > > include named, shared storage configurations.
> > > >
> > > > Thoughts?
> > > >
> > > > Robert
> > > >
> > > >
> > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > To recap the discussion from the Community Sync today: My
> > understanding
> > > > is
> > > > > that the consensus was:
> > > > >
> > > > > * Defer AuthZ aspects for later discussion.
> > > > >
> > > > > * Allow multiple Storage Config objects per Catalog. If more than
> one
> > > > > Storage Config is present, they are to have different (within the
> > > > catalog)
> > > > > names. Rishi to open a new PR for this (scoped to REST API + Sever
> > > > > changes).
> > > > >
> > > > > * Defer CLI changes until we have the full end-to-end system
> working
> > > (it
> > > > > should be usable via curl)
> > > > >
> > > > > * PR 4023 is on hold for now. It is to be rebased after adding
> > support
> > > > for
> > > > > multiple Storage Configs
> > > > >
> > > > > * The connection between storage names and storage access
> credentials
> > > is
> > > > to
> > > > > remain unchanged for now. It is effectively a deployment time
> concern
> > > > given
> > > > > the current OSS code.
> > > > >
> > > > > Please correct / add if I missed anything.
> > > > >
> > > > > From my POV, I think we still need to discuss how exactly entities
> > > > (tables,
> > > > > views) will link to Storage Config before we can resume PR 4023.
> > > > >
> > > > > I propose storing the config name in the Entity properties (not in
> > > Table
> > > > > Metadata properties).
> > > > >
> > > > > If clients also need a way to define it while interacting with
> > Polaris
> > > > over
> > > > > the IRC API, let's discuss how that can be done.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <
> dimas@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Srinivas,
> > > > > >
> > > > > > I see  that PR 4023 is active again. I'll try and post a fresh
> > review
> > > > > soon.
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > > > > sririshindra@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi JB and Dmitri,
> > > > > >>
> > > > > >> I raised PR #4023 <https://github.com/apache/polaris/pull/4023>
> > > > > yesterday
> > > > > >> as part of the broader per-table storage config effort, and I
> > > believe
> > > > it
> > > > > >> implements exactly what you are proposing.
> > > > > >>
> > > > > >> Building on the named storage configs introduced in PR #3409
> > > > > >> <https://github.com/apache/polaris/pull/3409>, this PR allows
> > > > > namespaces
> > > > > >> and tables to specify a polaris.storage.name property. This
> acts
> > as
> > > > the
> > > > > >> "identifier" JB mentioned, allowing credential vending to
> resolve
> > > the
> > > > > >> correct, named storage config at runtime without introducing new
> > > > > >> management
> > > > > >> APIs.
> > > > > >>
> > > > > >> Could you please take a look at the PR and let me know if this
> > > aligns
> > > > > with
> > > > > >> what you are looking for?
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Yufei Gu <fl...@gmail.com>]

Hi Adam,

A table can have more than one storage configuration without introducing a
separate relationship table. If we eventually need to support that use
case, we could model it as an array or model one storage config to contain
more than one storage. However, that would introduce additional complexity
around credential vending. e.g., what should happen if one storage backend
rejects the credential vending request while others succeed? How do we
compose multiple configs? What if the IAM roles from different config
conflict with each other? Can we combine multiple storage options into one
config or should we split them into different configs?

I think we can address those questions once we have concrete use cases. The
storage config data model itself may need an overhaul.The community may
also state, "Sorry, we don't want to support it because it is too
complicated," and provide certain workarounds to users.  I'm definitely
open to suggestions for this.

That said, I think we all agree that only one effective storage
configuration should apply at the table, namespace, or catalog level.
Otherwise, inheritance becomes difficult to define and reason about. If we
agree on that principle, I think it gives us a clear path forward.

Yufei


On Thu, Jun 18, 2026 at 4:42 AM Adam Szita <sz...@apache.org> wrote:

> Hi Yufei,
>
> Regarding #2:
> Would opting for a simple (assuming per table) storageConfigName or
> storageConfigId
> support future extensibility?
> Asking from a multiple-storage-per-table perspective as it's something that
> the Iceberg spec does allow IMHO.
>
> Cheers,
> Adam
>
> On Wed, 17 Jun 2026 at 00:37, Yufei Gu <fl...@gmail.com> wrote:
>
> > Hi folks, thanks for the discussion. They are really helpful!
> >
> > I think there are two separate questions here:
> >
> > 1. How should we store storage configurations? As catalog properties or
> as
> > separate entities?
> > I'd lean toward separate entities in the long term. It provides a cleaner
> > boundary between a catalog and its storage configurations, avoids
> bloating
> > the catalog object as the number of configs grows, and gives us a better
> > foundation for future access control since entity level permissions are
> > easier to reason about.
> >
> > 2. How should a table or namespace refer to a storage configuration?
> > I'd prefer storing a simple reference on the entity itself rather than
> > introducing a separate relationship table. Both approaches support
> > hierarchical lookup and inheritance when a storage config is not
> specified
> > at the current level. However, I don't think we should allow multiple
> > storage configs to be associated with a single table or namespace. Given
> > that cardinality, a simple storageConfigName or storageConfigId field
> seems
> > sufficient and easier to understand than a dedicated relationship model.
> >
> > For example:
> >
> > Catalog
> > ├─ entity: StorageConfigA
> > ├─ entity: StorageConfigB
> > └─ entity: StorageConfigC
> >
> > Table1 storage property -> StorageConfigA
> > Table2 storage property -> StorageConfigB
> > Table3 storage property -> StorageConfigC
> >
> > To me, this keeps the model simple while still supporting shared storage
> > configurations and future extensibility.
> >
> > Yufei
> >
> >
> > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Robert,
> > >
> > > The distinction between "attached" and "referenced" config makes sense.
> > >
> > > My impression from JB's message on Mar 19 in this thread [1] is that JB
> > is
> > > specifically proposing the "referenced" model, though.
> > >
> > > From my POV, both approaches can work, but the "referenced" model might
> > be
> > > more intuitive to users.
> > >
> > > [1] https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <sn...@snazy.de> wrote:
> > >
> > > > Hi all,
> > > >
> > > > AFAIK the current Polaris code appears to resolve the storage
> > > > configuration from the leaf (table), upwards via the namespace
> elements
> > > to
> > > > the catalog, choosing the first one found.
> > > > I am not sure whether that functionality is already used anywhere?
> > > > Being able to use a storage configuration on a namespace or a leaf
> > > appears
> > > > appealing.
> > > >
> > > > OTOH I see a demand for named storage configurations.
> > > >
> > > > I wonder whether the mechanism to derive the storage configuration
> > could
> > > > support both "directly attached" and "referenced" storage
> > configurations:
> > > > A namespace element or leaf can have either a "directly attached"
> > storage
> > > > configuration or a reference to a named one.
> > > >
> > > > We'd need management-ish APIs to handle the storage configurations
> for
> > a
> > > > namespace or leaf.
> > > > These APIs, which do not exist yet, could start with "directly
> > attached"
> > > > storage configurations and once the authz questions are resolved,
> > extend
> > > to
> > > > include named, shared storage configurations.
> > > >
> > > > Thoughts?
> > > >
> > > > Robert
> > > >
> > > >
> > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > To recap the discussion from the Community Sync today: My
> > understanding
> > > > is
> > > > > that the consensus was:
> > > > >
> > > > > * Defer AuthZ aspects for later discussion.
> > > > >
> > > > > * Allow multiple Storage Config objects per Catalog. If more than
> one
> > > > > Storage Config is present, they are to have different (within the
> > > > catalog)
> > > > > names. Rishi to open a new PR for this (scoped to REST API + Sever
> > > > > changes).
> > > > >
> > > > > * Defer CLI changes until we have the full end-to-end system
> working
> > > (it
> > > > > should be usable via curl)
> > > > >
> > > > > * PR 4023 is on hold for now. It is to be rebased after adding
> > support
> > > > for
> > > > > multiple Storage Configs
> > > > >
> > > > > * The connection between storage names and storage access
> credentials
> > > is
> > > > to
> > > > > remain unchanged for now. It is effectively a deployment time
> concern
> > > > given
> > > > > the current OSS code.
> > > > >
> > > > > Please correct / add if I missed anything.
> > > > >
> > > > > From my POV, I think we still need to discuss how exactly entities
> > > > (tables,
> > > > > views) will link to Storage Config before we can resume PR 4023.
> > > > >
> > > > > I propose storing the config name in the Entity properties (not in
> > > Table
> > > > > Metadata properties).
> > > > >
> > > > > If clients also need a way to define it while interacting with
> > Polaris
> > > > over
> > > > > the IRC API, let's discuss how that can be done.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <
> dimas@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Srinivas,
> > > > > >
> > > > > > I see  that PR 4023 is active again. I'll try and post a fresh
> > review
> > > > > soon.
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > > > > sririshindra@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi JB and Dmitri,
> > > > > >>
> > > > > >> I raised PR #4023 <https://github.com/apache/polaris/pull/4023>
> > > > > yesterday
> > > > > >> as part of the broader per-table storage config effort, and I
> > > believe
> > > > it
> > > > > >> implements exactly what you are proposing.
> > > > > >>
> > > > > >> Building on the named storage configs introduced in PR #3409
> > > > > >> <https://github.com/apache/polaris/pull/3409>, this PR allows
> > > > > namespaces
> > > > > >> and tables to specify a polaris.storage.name property. This
> acts
> > as
> > > > the
> > > > > >> "identifier" JB mentioned, allowing credential vending to
> resolve
> > > the
> > > > > >> correct, named storage config at runtime without introducing new
> > > > > >> management
> > > > > >> APIs.
> > > > > >>
> > > > > >> Could you please take a look at the PR and let me know if this
> > > aligns
> > > > > with
> > > > > >> what you are looking for?
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Adam Szita <sz...@apache.org>]

Hi Yufei,

Regarding #2:
Would opting for a simple (assuming per table) storageConfigName or
storageConfigId
support future extensibility?
Asking from a multiple-storage-per-table perspective as it's something that
the Iceberg spec does allow IMHO.

Cheers,
Adam

On Wed, 17 Jun 2026 at 00:37, Yufei Gu <fl...@gmail.com> wrote:

> Hi folks, thanks for the discussion. They are really helpful!
>
> I think there are two separate questions here:
>
> 1. How should we store storage configurations? As catalog properties or as
> separate entities?
> I'd lean toward separate entities in the long term. It provides a cleaner
> boundary between a catalog and its storage configurations, avoids bloating
> the catalog object as the number of configs grows, and gives us a better
> foundation for future access control since entity level permissions are
> easier to reason about.
>
> 2. How should a table or namespace refer to a storage configuration?
> I'd prefer storing a simple reference on the entity itself rather than
> introducing a separate relationship table. Both approaches support
> hierarchical lookup and inheritance when a storage config is not specified
> at the current level. However, I don't think we should allow multiple
> storage configs to be associated with a single table or namespace. Given
> that cardinality, a simple storageConfigName or storageConfigId field seems
> sufficient and easier to understand than a dedicated relationship model.
>
> For example:
>
> Catalog
> ├─ entity: StorageConfigA
> ├─ entity: StorageConfigB
> └─ entity: StorageConfigC
>
> Table1 storage property -> StorageConfigA
> Table2 storage property -> StorageConfigB
> Table3 storage property -> StorageConfigC
>
> To me, this keeps the model simple while still supporting shared storage
> configurations and future extensibility.
>
> Yufei
>
>
> On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Robert,
> >
> > The distinction between "attached" and "referenced" config makes sense.
> >
> > My impression from JB's message on Mar 19 in this thread [1] is that JB
> is
> > specifically proposing the "referenced" model, though.
> >
> > From my POV, both approaches can work, but the "referenced" model might
> be
> > more intuitive to users.
> >
> > [1] https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3
> >
> > Cheers,
> > Dmitri.
> >
> > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <sn...@snazy.de> wrote:
> >
> > > Hi all,
> > >
> > > AFAIK the current Polaris code appears to resolve the storage
> > > configuration from the leaf (table), upwards via the namespace elements
> > to
> > > the catalog, choosing the first one found.
> > > I am not sure whether that functionality is already used anywhere?
> > > Being able to use a storage configuration on a namespace or a leaf
> > appears
> > > appealing.
> > >
> > > OTOH I see a demand for named storage configurations.
> > >
> > > I wonder whether the mechanism to derive the storage configuration
> could
> > > support both "directly attached" and "referenced" storage
> configurations:
> > > A namespace element or leaf can have either a "directly attached"
> storage
> > > configuration or a reference to a named one.
> > >
> > > We'd need management-ish APIs to handle the storage configurations for
> a
> > > namespace or leaf.
> > > These APIs, which do not exist yet, could start with "directly
> attached"
> > > storage configurations and once the authz questions are resolved,
> extend
> > to
> > > include named, shared storage configurations.
> > >
> > > Thoughts?
> > >
> > > Robert
> > >
> > >
> > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > To recap the discussion from the Community Sync today: My
> understanding
> > > is
> > > > that the consensus was:
> > > >
> > > > * Defer AuthZ aspects for later discussion.
> > > >
> > > > * Allow multiple Storage Config objects per Catalog. If more than one
> > > > Storage Config is present, they are to have different (within the
> > > catalog)
> > > > names. Rishi to open a new PR for this (scoped to REST API + Sever
> > > > changes).
> > > >
> > > > * Defer CLI changes until we have the full end-to-end system working
> > (it
> > > > should be usable via curl)
> > > >
> > > > * PR 4023 is on hold for now. It is to be rebased after adding
> support
> > > for
> > > > multiple Storage Configs
> > > >
> > > > * The connection between storage names and storage access credentials
> > is
> > > to
> > > > remain unchanged for now. It is effectively a deployment time concern
> > > given
> > > > the current OSS code.
> > > >
> > > > Please correct / add if I missed anything.
> > > >
> > > > From my POV, I think we still need to discuss how exactly entities
> > > (tables,
> > > > views) will link to Storage Config before we can resume PR 4023.
> > > >
> > > > I propose storing the config name in the Entity properties (not in
> > Table
> > > > Metadata properties).
> > > >
> > > > If clients also need a way to define it while interacting with
> Polaris
> > > over
> > > > the IRC API, let's discuss how that can be done.
> > > >
> > > > WDYT?
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > > wrote:
> > > >
> > > > > Hi Srinivas,
> > > > >
> > > > > I see  that PR 4023 is active again. I'll try and post a fresh
> review
> > > > soon.
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > > > sririshindra@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi JB and Dmitri,
> > > > >>
> > > > >> I raised PR #4023 <https://github.com/apache/polaris/pull/4023>
> > > > yesterday
> > > > >> as part of the broader per-table storage config effort, and I
> > believe
> > > it
> > > > >> implements exactly what you are proposing.
> > > > >>
> > > > >> Building on the named storage configs introduced in PR #3409
> > > > >> <https://github.com/apache/polaris/pull/3409>, this PR allows
> > > > namespaces
> > > > >> and tables to specify a polaris.storage.name property. This acts
> as
> > > the
> > > > >> "identifier" JB mentioned, allowing credential vending to resolve
> > the
> > > > >> correct, named storage config at runtime without introducing new
> > > > >> management
> > > > >> APIs.
> > > > >>
> > > > >> Could you please take a look at the PR and let me know if this
> > aligns
> > > > with
> > > > >> what you are looking for?
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Yufei Gu <fl...@gmail.com>]

Hi folks, thanks for the discussion. They are really helpful!

I think there are two separate questions here:

1. How should we store storage configurations? As catalog properties or as
separate entities?
I'd lean toward separate entities in the long term. It provides a cleaner
boundary between a catalog and its storage configurations, avoids bloating
the catalog object as the number of configs grows, and gives us a better
foundation for future access control since entity level permissions are
easier to reason about.

2. How should a table or namespace refer to a storage configuration?
I'd prefer storing a simple reference on the entity itself rather than
introducing a separate relationship table. Both approaches support
hierarchical lookup and inheritance when a storage config is not specified
at the current level. However, I don't think we should allow multiple
storage configs to be associated with a single table or namespace. Given
that cardinality, a simple storageConfigName or storageConfigId field seems
sufficient and easier to understand than a dedicated relationship model.

For example:

Catalog
├─ entity: StorageConfigA
├─ entity: StorageConfigB
└─ entity: StorageConfigC

Table1 storage property -> StorageConfigA
Table2 storage property -> StorageConfigB
Table3 storage property -> StorageConfigC

To me, this keeps the model simple while still supporting shared storage
configurations and future extensibility.

Yufei


On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi Robert,
>
> The distinction between "attached" and "referenced" config makes sense.
>
> My impression from JB's message on Mar 19 in this thread [1] is that JB is
> specifically proposing the "referenced" model, though.
>
> From my POV, both approaches can work, but the "referenced" model might be
> more intuitive to users.
>
> [1] https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3
>
> Cheers,
> Dmitri.
>
> On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <sn...@snazy.de> wrote:
>
> > Hi all,
> >
> > AFAIK the current Polaris code appears to resolve the storage
> > configuration from the leaf (table), upwards via the namespace elements
> to
> > the catalog, choosing the first one found.
> > I am not sure whether that functionality is already used anywhere?
> > Being able to use a storage configuration on a namespace or a leaf
> appears
> > appealing.
> >
> > OTOH I see a demand for named storage configurations.
> >
> > I wonder whether the mechanism to derive the storage configuration could
> > support both "directly attached" and "referenced" storage configurations:
> > A namespace element or leaf can have either a "directly attached" storage
> > configuration or a reference to a named one.
> >
> > We'd need management-ish APIs to handle the storage configurations for a
> > namespace or leaf.
> > These APIs, which do not exist yet, could start with "directly attached"
> > storage configurations and once the authz questions are resolved, extend
> to
> > include named, shared storage configurations.
> >
> > Thoughts?
> >
> > Robert
> >
> >
> > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi All,
> > >
> > > To recap the discussion from the Community Sync today: My understanding
> > is
> > > that the consensus was:
> > >
> > > * Defer AuthZ aspects for later discussion.
> > >
> > > * Allow multiple Storage Config objects per Catalog. If more than one
> > > Storage Config is present, they are to have different (within the
> > catalog)
> > > names. Rishi to open a new PR for this (scoped to REST API + Sever
> > > changes).
> > >
> > > * Defer CLI changes until we have the full end-to-end system working
> (it
> > > should be usable via curl)
> > >
> > > * PR 4023 is on hold for now. It is to be rebased after adding support
> > for
> > > multiple Storage Configs
> > >
> > > * The connection between storage names and storage access credentials
> is
> > to
> > > remain unchanged for now. It is effectively a deployment time concern
> > given
> > > the current OSS code.
> > >
> > > Please correct / add if I missed anything.
> > >
> > > From my POV, I think we still need to discuss how exactly entities
> > (tables,
> > > views) will link to Storage Config before we can resume PR 4023.
> > >
> > > I propose storing the config name in the Entity properties (not in
> Table
> > > Metadata properties).
> > >
> > > If clients also need a way to define it while interacting with Polaris
> > over
> > > the IRC API, let's discuss how that can be done.
> > >
> > > WDYT?
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Srinivas,
> > > >
> > > > I see  that PR 4023 is active again. I'll try and post a fresh review
> > > soon.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > > sririshindra@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi JB and Dmitri,
> > > >>
> > > >> I raised PR #4023 <https://github.com/apache/polaris/pull/4023>
> > > yesterday
> > > >> as part of the broader per-table storage config effort, and I
> believe
> > it
> > > >> implements exactly what you are proposing.
> > > >>
> > > >> Building on the named storage configs introduced in PR #3409
> > > >> <https://github.com/apache/polaris/pull/3409>, this PR allows
> > > namespaces
> > > >> and tables to specify a polaris.storage.name property. This acts as
> > the
> > > >> "identifier" JB mentioned, allowing credential vending to resolve
> the
> > > >> correct, named storage config at runtime without introducing new
> > > >> management
> > > >> APIs.
> > > >>
> > > >> Could you please take a look at the PR and let me know if this
> aligns
> > > with
> > > >> what you are looking for?
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Robert,

The distinction between "attached" and "referenced" config makes sense.

My impression from JB's message on Mar 19 in this thread [1] is that JB is
specifically proposing the "referenced" model, though.

From my POV, both approaches can work, but the "referenced" model might be
more intuitive to users.

[1] https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3

Cheers,
Dmitri.

On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <sn...@snazy.de> wrote:

> Hi all,
>
> AFAIK the current Polaris code appears to resolve the storage
> configuration from the leaf (table), upwards via the namespace elements to
> the catalog, choosing the first one found.
> I am not sure whether that functionality is already used anywhere?
> Being able to use a storage configuration on a namespace or a leaf appears
> appealing.
>
> OTOH I see a demand for named storage configurations.
>
> I wonder whether the mechanism to derive the storage configuration could
> support both "directly attached" and "referenced" storage configurations:
> A namespace element or leaf can have either a "directly attached" storage
> configuration or a reference to a named one.
>
> We'd need management-ish APIs to handle the storage configurations for a
> namespace or leaf.
> These APIs, which do not exist yet, could start with "directly attached"
> storage configurations and once the authz questions are resolved, extend to
> include named, shared storage configurations.
>
> Thoughts?
>
> Robert
>
>
> On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > To recap the discussion from the Community Sync today: My understanding
> is
> > that the consensus was:
> >
> > * Defer AuthZ aspects for later discussion.
> >
> > * Allow multiple Storage Config objects per Catalog. If more than one
> > Storage Config is present, they are to have different (within the
> catalog)
> > names. Rishi to open a new PR for this (scoped to REST API + Sever
> > changes).
> >
> > * Defer CLI changes until we have the full end-to-end system working (it
> > should be usable via curl)
> >
> > * PR 4023 is on hold for now. It is to be rebased after adding support
> for
> > multiple Storage Configs
> >
> > * The connection between storage names and storage access credentials is
> to
> > remain unchanged for now. It is effectively a deployment time concern
> given
> > the current OSS code.
> >
> > Please correct / add if I missed anything.
> >
> > From my POV, I think we still need to discuss how exactly entities
> (tables,
> > views) will link to Storage Config before we can resume PR 4023.
> >
> > I propose storing the config name in the Entity properties (not in Table
> > Metadata properties).
> >
> > If clients also need a way to define it while interacting with Polaris
> over
> > the IRC API, let's discuss how that can be done.
> >
> > WDYT?
> >
> > Cheers,
> > Dmitri.
> >
> > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Srinivas,
> > >
> > > I see  that PR 4023 is active again. I'll try and post a fresh review
> > soon.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > sririshindra@gmail.com>
> > > wrote:
> > >
> > >> Hi JB and Dmitri,
> > >>
> > >> I raised PR #4023 <https://github.com/apache/polaris/pull/4023>
> > yesterday
> > >> as part of the broader per-table storage config effort, and I believe
> it
> > >> implements exactly what you are proposing.
> > >>
> > >> Building on the named storage configs introduced in PR #3409
> > >> <https://github.com/apache/polaris/pull/3409>, this PR allows
> > namespaces
> > >> and tables to specify a polaris.storage.name property. This acts as
> the
> > >> "identifier" JB mentioned, allowing credential vending to resolve the
> > >> correct, named storage config at runtime without introducing new
> > >> management
> > >> APIs.
> > >>
> > >> Could you please take a look at the PR and let me know if this aligns
> > with
> > >> what you are looking for?
> > >>
> > >
> >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Alexandre Dutra <ad...@apache.org>]

Hi all,

I like the distinction: attached vs referenced. I think we'd need new
management endpoints for both attached and referenced anyways.

Thanks,
Alex

On Mon, Jun 15, 2026 at 1:18 PM Robert Stupp <sn...@snazy.de> wrote:
>
> Hi all,
>
> AFAIK the current Polaris code appears to resolve the storage
> configuration from the leaf (table), upwards via the namespace elements to
> the catalog, choosing the first one found.
> I am not sure whether that functionality is already used anywhere?
> Being able to use a storage configuration on a namespace or a leaf appears
> appealing.
>
> OTOH I see a demand for named storage configurations.
>
> I wonder whether the mechanism to derive the storage configuration could
> support both "directly attached" and "referenced" storage configurations:
> A namespace element or leaf can have either a "directly attached" storage
> configuration or a reference to a named one.
>
> We'd need management-ish APIs to handle the storage configurations for a
> namespace or leaf.
> These APIs, which do not exist yet, could start with "directly attached"
> storage configurations and once the authz questions are resolved, extend to
> include named, shared storage configurations.
>
> Thoughts?
>
> Robert
>
>
> On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > To recap the discussion from the Community Sync today: My understanding is
> > that the consensus was:
> >
> > * Defer AuthZ aspects for later discussion.
> >
> > * Allow multiple Storage Config objects per Catalog. If more than one
> > Storage Config is present, they are to have different (within the catalog)
> > names. Rishi to open a new PR for this (scoped to REST API + Sever
> > changes).
> >
> > * Defer CLI changes until we have the full end-to-end system working (it
> > should be usable via curl)
> >
> > * PR 4023 is on hold for now. It is to be rebased after adding support for
> > multiple Storage Configs
> >
> > * The connection between storage names and storage access credentials is to
> > remain unchanged for now. It is effectively a deployment time concern given
> > the current OSS code.
> >
> > Please correct / add if I missed anything.
> >
> > From my POV, I think we still need to discuss how exactly entities (tables,
> > views) will link to Storage Config before we can resume PR 4023.
> >
> > I propose storing the config name in the Entity properties (not in Table
> > Metadata properties).
> >
> > If clients also need a way to define it while interacting with Polaris over
> > the IRC API, let's discuss how that can be done.
> >
> > WDYT?
> >
> > Cheers,
> > Dmitri.
> >
> > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Srinivas,
> > >
> > > I see  that PR 4023 is active again. I'll try and post a fresh review
> > soon.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> > sririshindra@gmail.com>
> > > wrote:
> > >
> > >> Hi JB and Dmitri,
> > >>
> > >> I raised PR #4023 <https://github.com/apache/polaris/pull/4023>
> > yesterday
> > >> as part of the broader per-table storage config effort, and I believe it
> > >> implements exactly what you are proposing.
> > >>
> > >> Building on the named storage configs introduced in PR #3409
> > >> <https://github.com/apache/polaris/pull/3409>, this PR allows
> > namespaces
> > >> and tables to specify a polaris.storage.name property. This acts as the
> > >> "identifier" JB mentioned, allowing credential vending to resolve the
> > >> correct, named storage config at runtime without introducing new
> > >> management
> > >> APIs.
> > >>
> > >> Could you please take a look at the PR and let me know if this aligns
> > with
> > >> what you are looking for?
> > >>
> > >
> >

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Robert Stupp <sn...@snazy.de>]

Hi all,

AFAIK the current Polaris code appears to resolve the storage
configuration from the leaf (table), upwards via the namespace elements to
the catalog, choosing the first one found.
I am not sure whether that functionality is already used anywhere?
Being able to use a storage configuration on a namespace or a leaf appears
appealing.

OTOH I see a demand for named storage configurations.

I wonder whether the mechanism to derive the storage configuration could
support both "directly attached" and "referenced" storage configurations:
A namespace element or leaf can have either a "directly attached" storage
configuration or a reference to a named one.

We'd need management-ish APIs to handle the storage configurations for a
namespace or leaf.
These APIs, which do not exist yet, could start with "directly attached"
storage configurations and once the authz questions are resolved, extend to
include named, shared storage configurations.

Thoughts?

Robert


On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> To recap the discussion from the Community Sync today: My understanding is
> that the consensus was:
>
> * Defer AuthZ aspects for later discussion.
>
> * Allow multiple Storage Config objects per Catalog. If more than one
> Storage Config is present, they are to have different (within the catalog)
> names. Rishi to open a new PR for this (scoped to REST API + Sever
> changes).
>
> * Defer CLI changes until we have the full end-to-end system working (it
> should be usable via curl)
>
> * PR 4023 is on hold for now. It is to be rebased after adding support for
> multiple Storage Configs
>
> * The connection between storage names and storage access credentials is to
> remain unchanged for now. It is effectively a deployment time concern given
> the current OSS code.
>
> Please correct / add if I missed anything.
>
> From my POV, I think we still need to discuss how exactly entities (tables,
> views) will link to Storage Config before we can resume PR 4023.
>
> I propose storing the config name in the Entity properties (not in Table
> Metadata properties).
>
> If clients also need a way to define it while interacting with Polaris over
> the IRC API, let's discuss how that can be done.
>
> WDYT?
>
> Cheers,
> Dmitri.
>
> On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Srinivas,
> >
> > I see  that PR 4023 is active again. I'll try and post a fresh review
> soon.
> >
> > Cheers,
> > Dmitri.
> >
> > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <
> sririshindra@gmail.com>
> > wrote:
> >
> >> Hi JB and Dmitri,
> >>
> >> I raised PR #4023 <https://github.com/apache/polaris/pull/4023>
> yesterday
> >> as part of the broader per-table storage config effort, and I believe it
> >> implements exactly what you are proposing.
> >>
> >> Building on the named storage configs introduced in PR #3409
> >> <https://github.com/apache/polaris/pull/3409>, this PR allows
> namespaces
> >> and tables to specify a polaris.storage.name property. This acts as the
> >> "identifier" JB mentioned, allowing credential vending to resolve the
> >> correct, named storage config at runtime without introducing new
> >> management
> >> APIs.
> >>
> >> Could you please take a look at the PR and let me know if this aligns
> with
> >> what you are looking for?
> >>
> >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

To recap the discussion from the Community Sync today: My understanding is
that the consensus was:

* Defer AuthZ aspects for later discussion.

* Allow multiple Storage Config objects per Catalog. If more than one
Storage Config is present, they are to have different (within the catalog)
names. Rishi to open a new PR for this (scoped to REST API + Sever changes).

* Defer CLI changes until we have the full end-to-end system working (it
should be usable via curl)

* PR 4023 is on hold for now. It is to be rebased after adding support for
multiple Storage Configs

* The connection between storage names and storage access credentials is to
remain unchanged for now. It is effectively a deployment time concern given
the current OSS code.

Please correct / add if I missed anything.

From my POV, I think we still need to discuss how exactly entities (tables,
views) will link to Storage Config before we can resume PR 4023.

I propose storing the config name in the Entity properties (not in Table
Metadata properties).

If clients also need a way to define it while interacting with Polaris over
the IRC API, let's discuss how that can be done.

WDYT?

Cheers,
Dmitri.

On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi Srinivas,
>
> I see  that PR 4023 is active again. I'll try and post a fresh review soon.
>
> Cheers,
> Dmitri.
>
> On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <sr...@gmail.com>
> wrote:
>
>> Hi JB and Dmitri,
>>
>> I raised PR #4023 <https://github.com/apache/polaris/pull/4023> yesterday
>> as part of the broader per-table storage config effort, and I believe it
>> implements exactly what you are proposing.
>>
>> Building on the named storage configs introduced in PR #3409
>> <https://github.com/apache/polaris/pull/3409>, this PR allows namespaces
>> and tables to specify a polaris.storage.name property. This acts as the
>> "identifier" JB mentioned, allowing credential vending to resolve the
>> correct, named storage config at runtime without introducing new
>> management
>> APIs.
>>
>> Could you please take a look at the PR and let me know if this aligns with
>> what you are looking for?
>>
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Srinivas,

I see  that PR 4023 is active again. I'll try and post a fresh review soon.

Cheers,
Dmitri.

On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra <sr...@gmail.com>
wrote:

> Hi JB and Dmitri,
>
> I raised PR #4023 <https://github.com/apache/polaris/pull/4023> yesterday
> as part of the broader per-table storage config effort, and I believe it
> implements exactly what you are proposing.
>
> Building on the named storage configs introduced in PR #3409
> <https://github.com/apache/polaris/pull/3409>, this PR allows namespaces
> and tables to specify a polaris.storage.name property. This acts as the
> "identifier" JB mentioned, allowing credential vending to resolve the
> correct, named storage config at runtime without introducing new management
> APIs.
>
> Could you please take a look at the PR and let me know if this aligns with
> what you are looking for?
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Srinivas,

Thank you for the update. I will review the PR, as it appears to be
perfectly aligned with the use case we discussed.

Regards,
JB

On Thu, Mar 19, 2026 at 10:42 AM Srinivas Rishindra <sr...@gmail.com>
wrote:

> Hi JB and Dmitri,
>
> I raised PR #4023 <https://github.com/apache/polaris/pull/4023> yesterday
> as part of the broader per-table storage config effort, and I believe it
> implements exactly what you are proposing.
>
> Building on the named storage configs introduced in PR #3409
> <https://github.com/apache/polaris/pull/3409>, this PR allows namespaces
> and tables to specify a polaris.storage.name property. This acts as the
> "identifier" JB mentioned, allowing credential vending to resolve the
> correct, named storage config at runtime without introducing new management
> APIs.
>
> Could you please take a look at the PR and let me know if this aligns with
> what you are looking for?
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Srinivas Rishindra <sr...@gmail.com>]

Hi JB and Dmitri,

I raised PR #4023 <https://github.com/apache/polaris/pull/4023> yesterday
as part of the broader per-table storage config effort, and I believe it
implements exactly what you are proposing.

Building on the named storage configs introduced in PR #3409
<https://github.com/apache/polaris/pull/3409>, this PR allows namespaces
and tables to specify a polaris.storage.name property. This acts as the
"identifier" JB mentioned, allowing credential vending to resolve the
correct, named storage config at runtime without introducing new management
APIs.

Could you please take a look at the PR and let me know if this aligns with
what you are looking for?

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Dmitri and Srinivas,

That is exactly the same context. I wanted to provide some additional
perspective by focusing specifically on the use case.

I also prefer managing the storage configurations at the catalog level,
which was the intent behind the "identifier" idea. I will review the PRs
mentioned and follow up.

Regards,
JB

On Thu, Mar 19, 2026 at 12:12 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi JB,
>
> This looks related to the per-table storage config discussion [1], right?
>
> A name for StorageConfigInfo was already introduced in [3409]. I suppose,
> we could build on that and the next step would be to adjust the lookup
> mechanism to consider storage config names.
>
> Overall, I think this idea is pretty straight-forward and can likely fully
> implement the non-credential aspect of the per-table storage config
> proposal [1].
>
> I tend to faviour managing named storage configs at the catalog level over
> unnamed config at arbitrary places of the catalog tree. I think the former
> is simpler from the user's perspective and functionally equivalent to the
> latter.
>
> [1] https://lists.apache.org/thread/boqdzdtmhhk1bncv2xr43sz4nsrhgwro
>
> [3409] https://github.com/apache/polaris/pull/3409
>
> Cheers,
> Dmitri.
>
> On Wed, Mar 18, 2026 at 11:19 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi folks,
> >
> > I would like to discuss a use case and potentially an enhancement on
> > Polaris.
> >
> > The use case is as follows:
> > 1. I have one Polaris catalog.
> > 2. I have a S3 StorageConfigurationInfo (containing my S3 credentials,
> > allowed locations, region, endpoint).
> > 3. Optionally, at namespace level, I can have base-location, inheriting
> the
> > catalog's storage credentials.
> > 4. A table level, I can specify a location/base-location, again
> inheriting
> > the catalog's storage credentials.
> > It means that a table can have its data in s3://bucket-a/table1/, and
> > another table can use s3://bucket-b/table2/, as long as both buckets are
> > accessible with the same catalog-level credentials and are in the
> catalog's
> > allowed locations.
> >
> > Now, I would like to have (again in a single catalog), one table on a S3,
> > another table on another S3.
> > The credential vending walks up the entity hierarchy
> > (FileIOUtil.findStorageInfoFromHierarchy) and uses the first
> > StorageConfigurationInfo it finds, which is always at the catalog level.
> >
> > The only viable option today is to create separate catalogs, each with
> its
> > own S3 storage configuration pointing to a different bucket.
> >
> > I would like to introduce an "identifier" on StorageConfigurationInfo and
> > having several StorageConfigurationInfos defined at catalog level.
> > Then, we could add an extra table property to define the
> > StorageConfigurationInfo identifier to us.
> >
> > Thoughts ?
> >
> > I would be happy to work on a proposal about that if there is interest.
> >
> > Regards
> > JB
> >
>

Re: [DISCUSS] Multiple StorageConfigurationInfos per catalog

[Dmitri Bourlatchkov <di...@apache.org>]

Hi JB,

This looks related to the per-table storage config discussion [1], right?

A name for StorageConfigInfo was already introduced in [3409]. I suppose,
we could build on that and the next step would be to adjust the lookup
mechanism to consider storage config names.

Overall, I think this idea is pretty straight-forward and can likely fully
implement the non-credential aspect of the per-table storage config
proposal [1].

I tend to faviour managing named storage configs at the catalog level over
unnamed config at arbitrary places of the catalog tree. I think the former
is simpler from the user's perspective and functionally equivalent to the
latter.

[1] https://lists.apache.org/thread/boqdzdtmhhk1bncv2xr43sz4nsrhgwro

[3409] https://github.com/apache/polaris/pull/3409

Cheers,
Dmitri.

On Wed, Mar 18, 2026 at 11:19 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Hi folks,
>
> I would like to discuss a use case and potentially an enhancement on
> Polaris.
>
> The use case is as follows:
> 1. I have one Polaris catalog.
> 2. I have a S3 StorageConfigurationInfo (containing my S3 credentials,
> allowed locations, region, endpoint).
> 3. Optionally, at namespace level, I can have base-location, inheriting the
> catalog's storage credentials.
> 4. A table level, I can specify a location/base-location, again inheriting
> the catalog's storage credentials.
> It means that a table can have its data in s3://bucket-a/table1/, and
> another table can use s3://bucket-b/table2/, as long as both buckets are
> accessible with the same catalog-level credentials and are in the catalog's
> allowed locations.
>
> Now, I would like to have (again in a single catalog), one table on a S3,
> another table on another S3.
> The credential vending walks up the entity hierarchy
> (FileIOUtil.findStorageInfoFromHierarchy) and uses the first
> StorageConfigurationInfo it finds, which is always at the catalog level.
>
> The only viable option today is to create separate catalogs, each with its
> own S3 storage configuration pointing to a different bucket.
>
> I would like to introduce an "identifier" on StorageConfigurationInfo and
> having several StorageConfigurationInfos defined at catalog level.
> Then, we could add an extra table property to define the
> StorageConfigurationInfo identifier to us.
>
> Thoughts ?
>
> I would be happy to work on a proposal about that if there is interest.
>
> Regards
> JB
>