You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by Dmitri Bourlatchkov <di...@apache.org> on 2026/09/08 22:49:46 UTC
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Hi All,
My impression from this thread is that it might be time for a call to
discuss all the related issues and try to arrive at a shared implementation
plan.
I know meetings are not ideal, but in this case it might be beneficial as a
means for achieving a common understanding of the set of problems and
priorities related to this thread. A dedicated metrics meeting worked well
from my POV.
Allocating a time slot in the next community meeting might be an option,
although I think we might need the full hour in this case.
WDYT?
Thanks,
Dmitri.
On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <pr...@gmail.com>
wrote:
> Hi all,
>
> I rewrote https://github.com/apache/polaris/pull/5035, following the
> discussion to focus only on the SPI foundation.
>
> What is in the PR now:
>
> - A written manager-level consistency contract in
> site/content/in-dev/unreleased/persistence-consistency.md.
> - CAS-aware EntityMutation / GrantMutation records.
> - MetaStoreChangeSet and BasePersistence#commitChangeSet, with backends
> opting in via supportsAtomicMixedCommit().
> - commitChangeSet implementations for the in-memory TreeMap backend and
> JDBC.
> - AtomicOperationMetaStoreManager and TransactionalMetaStoreManagerImpl
> grant/revoke paths now build one change set per operation, falling back
> to
> individual operations when the backend does not support mixed commits.
> - Tests for the change-set API and TreeMap atomic commits.
>
> I intentionally left out:
>
> - Entity deletes in MetaStoreChangeSet.
> - Refactoring createCatalog, dropEntity, and renameEntity to use change
> sets.
> - NoSQL support for commitChangeSet.
>
> I am treating this as Phase 1 so the contract and the SPI primitive can be
> reviewed before the larger operation migrations land.
>
> Could you please take a look? In particular, I would like to know whether
> the contract captures the consensus so far, or if it needs to address retry
> / external-work coordination before we merge this foundation.
>
> Thanks,
> Prithvi S
>
> On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Robert,
> >
> > I agree that the client-visible operation (e.g. REST API request)
> consists
> > of many distinct phases. The database / persistence changes are just one
> of
> > these phases. STS (as an example) is another. My point about transactions
> > in JDBC Persistence applies to the former (database changes) phase - one
> > retry attempt there should ideally involve exactly one transaction.
> >
> > We certainly need logic inside Polaris Servers to to coordinate requests
> to
> > various external systems depending on outcomes from previous request
> > processing phases.
> >
> > I also agree that validation based on database state should be repeated
> > from scratch if we retry the database changes.
> >
> > However, internal RBAC authorization naturally has to happen inside the
> > database update phase (read-check-write). IIRC, that is part of the
> > implicit consistency guarantees, which were discussed during early
> project
> > months (but never got written down in full clarity, unfortunately). If
> the
> > Authorization check is inside that phase for internal RBAC, I suppose it
> > will be there for other Authorizers too and that can involve external
> > systems (e.g. OPA / Ranger). We can certainly expect authorization to be
> > efficient, but I'm not sure how quick it can be in practice. There's
> > certainly potential for delays in some (perhaps infrequent) cases.
> >
> > Cheers,
> > Dmitri.
> >
> >
> > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <sn...@snazy.de> wrote:
> >
> > > Hi Dmitri,
> > >
> > > I agree that one logical operation needs a consistent view of the whole
> > > backend state.
> > > What worries me is treating one long JDBC transaction as the boundary
> of
> > > that operation and then retrying the complete request if the commit
> > fails.
> > >
> > > The request can also write metadata to object storage or call STS.
> > > Each system can leave us with an outcome whose certainty we cannot
> know.
> > > The database may have committed before the connection was lost, an
> > > object-store write may have succeeded before a timeout, or STS may have
> > > issued credentials before its response was lost.
> > > A database rollback cannot undo any of those effects, and retrying the
> > > whole request may repeat them.
> > >
> > > So I think the contract has to separate the overall operation from each
> > > attempt to commit it.
> > > Each database or backend commit attempt should stay short.
> > > When we retry, validation and authorization need to use the state for
> > that
> > > new attempt, and we need clear rules for which external work can be
> > reused,
> > > repeated, cleaned up, or reconciled after an uncertain outcome.
> > >
> > > Some contextual state could be useful for carrying that state.
> > > But the JDBC connection and transaction should be an implementation
> > detail
> > > inside it, not the boundary of the complete operation.
> > >
> > > Cheers,
> > > Robert
> > >
> > >
> > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > The idea of a backend-agnostic change-set primitive sounds good.
> > > >
> > > > However, I am not sure it is sufficient for all the use cases we
> > touched
> > > > here. More specifically, the state read by validation code is not
> > > > necessarily reflected in the change set:
> > > >
> > > > * Unchanged entities might be considered by validation, but changed
> in
> > a
> > > > parallel request.
> > > > * Even for changed entities the state read by validation may differ
> > from
> > > > the state being altered. Some validation code talks to the MetaStore
> > > > directly, outside of the data produced by the Resolver.
> > > >
> > > > I do not think Polaris offers any explicit mechanisms (ATM) to ensure
> > > > consistency between these reads and subsequent writes.
> > > >
> > > > As far as JDBC goes, running an overarching Transaction across all
> > reads
> > > > and writes in the same request, at the SERIALIZABLE isolation level
> in
> > > the
> > > > backing RDBMS could solve the problem, I think.
> > > >
> > > > Indeed, such a transaction might be long to accommodate calls made by
> > > > Polaris to external storage, etc. However, is that a problem? I think
> > the
> > > > alternative is for the JDBC Persistence impl. to "manually" track all
> > > reads
> > > > under the same request and redo them in the small transaction that
> > > persists
> > > > the writes. This will also add RDBMS overhead and require complex
> code
> > in
> > > > Polaris to handle the data properly.
> > > >
> > > > Tracking the request-wide transaction can be done only for JDBC
> without
> > > > leaking "transaction" concepts to the NoSQL Persistence, I think.
> NoSQL
> > > > will use other mechanisms to ensure read/write consistency.
> > > >
> > > > Connecting to Robert's email (a parallel branch in this discussion),
> > I'd
> > > > like to propose this approach:
> > > >
> > > > * Each request establishes a "Data Context"
> > > > - In JDBC the Data Context corresponds to a JDBC Connection + Tx
> > > > - In NoSQL the Data Context tracks one or more reference hashes
> > > > * All Persistence access in the same request goes through the same
> Data
> > > > Context
> > > > * All Persistence changes are committed once at the end of the
> request
> > > > - Not all changes have to be globally atomic. For example, changes
> in
> > > > Catalogs A and B do not have to be atomic with respect to each other.
> > We
> > > > can go deeper into this later. This is relevant to NoSQL.
> > > > - Transactional backends like JDBC can, of course, choose to make
> all
> > > > changes globally atomic.
> > > > * A commit can fail in two main ways:
> > > > - A retriable failure like an optimistic lock error or Tx
> > > serializability
> > > > error
> > > > - A non-triable logical error (e.g. entity not found)
> > > > * On a retriable error the whole request is re-attempted (as if
> > > resubmitted
> > > > by a client) a few times (configurable timeout).
> > > >
> > > > WDYT?
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré <
> jb@nanthrax.net
> > >
> > > > wrote:
> > > >
> > > > > Hi all
> > > > >
> > > > > I think Robert has a good point.
> > > > >
> > > > > If the atomicity guarantee lives only in BasePersistence, then the
> > > > > manager contract can't tell a caller whether the state it read for
> > > > > validation/authorization/credential-vending is the same state that
> > > > > eventually commits. That's the actual bug class behind the JDBC
> > > > > symptoms (not any single operation being non-atomic, but the
> contract
> > > > > being silent about it. Every operation-specific fix (like #4939,
> > #5035
> > > > > or #5095) re-answers this question locally and it keeps recurring.
> So
> > > > > the deliverable should start with a written consistency contract at
> > > > > the manager level.
> > > > >
> > > > > I'm not sure migrating everyone to
> TransactionalMetaStoreManagerImpl
> > > > > is a good idea. It would tie the logical change set to a DB
> > > > > transaction spanning the REST request. It means:
> > > > > - it holds a durable transaction open across slow external work
> > > > > (credential vending, OPA, Ranger, ...)
> > > > > - it doesn't map to NoSQL
> > > > > - It wraps single-row updates in runWiithinTransaction, which is a
> > > > overhead
> > > > >
> > > > > So, I think the transactional manager isn't a portable target. It's
> > > > > "only" one backend's strategy.
> > > > >
> > > > > I think Privthi's approach is right. A backend-agnostic change-set
> > > > > primitive with a documented fallback is the correct shape. There is
> > > > > one caveat: without carrying the original entitiy (not just the new
> > > > > one) the change set can't express optimistic concurrency.
> > > > >
> > > > > I propose the following multi-steps approach:
> > > > > 1. We write the manager-level consistency contract as Robert asked.
> > It
> > > > > should include the explicit statement that a logical change set is
> > not
> > > > > a request-scope DB transaction.
> > > > > 2. We make Compare And Swap (optimistic-concurrency pattern)
> baseline
> > > > > first-class in EntityMutation before merging the SPI
> > > > > 3. Then we refactor createCatalog/dropEntity/renameEntity.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <sn...@snazy.de>
> wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Yufei's clarification about where the atomicity guarantee is
> > defined
> > > > > seems
> > > > > > important.
> > > > > > If it is a property of the lower-level BasePersistence contract
> > > rather
> > > > > than
> > > > > > the general PolarisMetaStoreManager contract, the general
> contract
> > > does
> > > > > not
> > > > > > tell callers whether the state used for validation,
> authorization,
> > or
> > > > > > credential vending is consistent with the change that eventually
> > > > commits.
> > > > > >
> > > > > > The current PRs suggest that operation-specific multi-object
> > methods
> > > > can
> > > > > > fix individual cases, while leaving the same contract question to
> > > recur
> > > > > for
> > > > > > each new case.
> > > > > >
> > > > > > I would also avoid defining a logical change set as a database
> > > > > transaction
> > > > > > around an entire REST request.
> > > > > > That would tie the contract to the backend and database, and
> could
> > > keep
> > > > > the
> > > > > > durable attempt open across slow external work.
> > > > > >
> > > > > > So is the choice really between the two current manager
> > > > implementations,
> > > > > or
> > > > > > do we first need to revisit the boundary and guarantees exposed
> to
> > > > their
> > > > > > callers?
> > > > > >
> > > > > > Cheers,
> > > > > > Robert
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov <
> > > dimas@apache.org
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Yufei,
> > > > > > >
> > > > > > > Thanks for the link. I stand corrected. The "one atomic change
> > per
> > > > > method"
> > > > > > > contract as defined in javadoc does apply only to
> BasePersistence
> > > > > > > and AtomicOperationMetaStoreManager (which delegates to
> > > > > BasePersistence).
> > > > > > >
> > > > > > > Note that all non-test Persistence implementations in the
> Polaris
> > > > > codebase
> > > > > > > extend those classes (which is probably why I was confused
> about
> > > > > atomicity
> > > > > > > expectations).
> > > > > > >
> > > > > > > However, this creates a gap in the Persistence SPI
> specification.
> > > If
> > > > > > > other PolarisMetaStoreManager implementations do not have to
> > comply
> > > > > with
> > > > > > > this principle, it will create a conceptual difficulty at call
> > > sites.
> > > > > How
> > > > > > > can PolarisMetaStoreManager callers reason about consistency
> and
> > > > > durability
> > > > > > > behaviours in general?
> > > > > > >
> > > > > > > I believe we need to address that as part of this discussion.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <flyrain000@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > > The MetaStore SPI is currently defined with the idea that
> one
> > > > > method
> > > > > > > call
> > > > > > > > means one atomic change.
> > > > > > > >
> > > > > > > > If the "MetaStore SPI" refers to the interface
> > > > > PolarisMetaStoreManager, I
> > > > > > > > don't think we've ever state each method to be atomic. We did
> > > > clarify
> > > > > > > > atomicity[1] in the interface BasePersistence though.
> > > > > > > >
> > > > > > > >
> > > > > > > > 1.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Yufei
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri Bourlatchkov <
> > > > > dimas@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Yufei,
> > > > > > > > >
> > > > > > > > > I agree that JDBC transactions must be handled more
> > explicitly.
> > > > > > > However,
> > > > > > > > > I'm not sure that simply moving to
> > > > > TransactionalMetaStoreManagerImpl is
> > > > > > > > > sufficient.
> > > > > > > > >
> > > > > > > > > The MetaStore SPI is currently defined with the idea that
> one
> > > > > method
> > > > > > > call
> > > > > > > > > means one atomic change [1]. The "transactional" MetaStore
> > > impl.
> > > > is
> > > > > > > but a
> > > > > > > > > sub-case of that. It cannot alter the high-level contract.
> > > > > > > > >
> > > > > > > > > We could add SPI methods having multiple object parameters
> to
> > > > > represent
> > > > > > > > > grouped changes, but I am not sure it will be a sound
> design.
> > > > This
> > > > > will
> > > > > > > > > bloat the interface surfaces and require extra impl. effort
> > for
> > > > > each
> > > > > > > > > backend type. More importantly, adding multi-arg change
> > methods
> > > > > still
> > > > > > > > won't
> > > > > > > > > address the problem of reads being consistent with writes,
> > > > because
> > > > > each
> > > > > > > > > method call will still be independent regarding the data
> > stored
> > > > in
> > > > > the
> > > > > > > > > database.
> > > > > > > > >
> > > > > > > > > I tend to think we need to introduce a "change set" or
> > "atomic
> > > > > batch"
> > > > > > > > > concept to core Persistence and associate each REST API
> > request
> > > > > with
> > > > > > > one
> > > > > > > > > such change set, which will be committed (or rolled back)
> at
> > > the
> > > > > end of
> > > > > > > > the
> > > > > > > > > request. I believe Ayush mentioned a similar concept in PR
> > 4939
> > > > > [2]. In
> > > > > > > > > JDBC each change set will naturally be associated with an
> > RDBMS
> > > > > > > > > transaction. In NoSQL persistence, each atomic change set
> > will
> > > be
> > > > > > > > > associated with one CAS operation on the underlying
> database.
> > > > > > > > >
> > > > > > > > > [1]
> > > > > https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
> > > > > > > > >
> > > > > > > > > [2]
> > > > > https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
> > flyrain000@gmail.com
> > > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for raising this, Dmitri.
> > > > > > > > > >
> > > > > > > > > > These are valid concerns, and they were already
> recognized
> > > when
> > > > > we
> > > > > > > > > > introduced JDBC persistence to Polaris. At that time, we
> > > chose
> > > > > to use
> > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC due to the
> > > > > simplicity. I
> > > > > > > > > think
> > > > > > > > > > most of the issues mentioned here can already be
> addressed
> > by
> > > > > > > > > > TransactionalMetaStoreManagerImpl.
> > > > > > > > > >
> > > > > > > > > > For example, rename is already wrapped in a transaction
> in
> > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. Similarly, catalog
> > > > > creation,
> > > > > > > > which
> > > > > > > > > > involves reading and creating multiple objects, is also
> > > > executed
> > > > > > > > within a
> > > > > > > > > > transaction [2].
> > > > > > > > > >
> > > > > > > > > > I see two possible directions:
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > >
> > > > > > > > > > Modify AtomicOperationMetaStoreManager together with
> the
> > > > > > > persistence
> > > > > > > > > > backends (such as JDBC) to provide the required
> > > consistency
> > > > > > > > guarantees
> > > > > > > > > > for
> > > > > > > > > > specific operations, similar to what
> > > > > > > > TransactionalMetaStoreManagerImpl
> > > > > > > > > > does.
> > > > > > > > > > 2.
> > > > > > > > > >
> > > > > > > > > > Migrate the persistence backends (such as JDBC) to use
> > > > > > > > > > TransactionalMetaStoreManagerImpl directly. We may
> have
> > to
> > > > > deal
> > > > > > > with
> > > > > > > > > > transactional semantic mismatches across different
> > > > persistence
> > > > > > > > > backends.
> > > > > > > > > > For example, we would likely avoid using JDBC's
> > > > > > > > `runWithinTransaction`
> > > > > > > > > > for
> > > > > > > > > > single row updates, which adds additional overhead and
> > > > > complexity
> > > > > > > > > > without
> > > > > > > > > > benefits.
> > > > > > > > > >
> > > > > > > > > > References:
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > > > > > > > > > 2.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > > > > > > > > >
> > > > > > > > > > Yufei
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri Bourlatchkov <
> > > > > > > dimas@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > Ayush and Prithvi recently contributed a couple of
> > > > interesting
> > > > > PRs:
> > > > > > > > > > > [4939], [5035].
> > > > > > > > > > >
> > > > > > > > > > > It looks like people are starting to encounter
> > consistency
> > > > > issues
> > > > > > > in
> > > > > > > > > > > JDBC persistence.
> > > > > > > > > > >
> > > > > > > > > > > The PRs provide valuable insight into the underlying
> > > issues.
> > > > > They
> > > > > > > > offer
> > > > > > > > > > > incremental fixes that can work. However, I believe it
> is
> > > > time
> > > > > for
> > > > > > > > the
> > > > > > > > > > > Polaris community to review and improve this area of
> the
> > > > > codebase
> > > > > > > > > > > holistically.
> > > > > > > > > > >
> > > > > > > > > > > By this, I mean finding a solution that can be applied
> to
> > > all
> > > > > > > > > > > persistence backends (in-memory, JDBC, NoSQL) and
> > addresses
> > > > > these
> > > > > > > > > > > aspects:
> > > > > > > > > > >
> > > > > > > > > > > * Supporting concurrent and consistent changes where
> the
> > > > > service
> > > > > > > > reads
> > > > > > > > > > > and validates current catalog state, then commits a
> > > change
> > > > > (e.g.
> > > > > > > > > > > name clashes during renames).
> > > > > > > > > > > * Supporting consistent but independent changes to RBAC
> > > > grants
> > > > > and
> > > > > > > > > > > MetaStore entities. This independence is needed to
> > > support
> > > > > > > > > > > external authorizers like OPA and Ranger.
> > > > > > > > > > > * Supporting atomic changes across multiple similar
> > > entities.
> > > > > > > > > > > * Supporting authorization-based filtering of list
> > > operations
> > > > > (cf.
> > > > > > > > > > > [4831]).
> > > > > > > > > > > * Supporting credential-vending decisions that are
> rooted
> > > in
> > > > > the
> > > > > > > > > > > exact state of the catalog.
> > > > > > > > > > > * Supporting server-side retries for transient
> > persistence
> > > > > failures
> > > > > > > > > > > (e.g. RDBMS Tx serializability failures).
> > > > > > > > > > >
> > > > > > > > > > > Please share your comments and ideas.
> > > > > > > > > > >
> > > > > > > > > > > [4831] https://github.com/apache/polaris/pull/4831
> > > > > > > > > > >
> > > > > > > > > > > [4939] https://github.com/apache/polaris/pull/4939
> > > > > > > > > > >
> > > > > > > > > > > [5035] https://github.com/apache/polaris/pull/5035
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Dmitri
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi EJ,
(replying separately to each item)
Very good point about "business rules" (although I personally do not like
the term at all :) )
I believe this area of improvement is actually independent of the bigger
Persistence discussion. If there's interest in refactoring that code to
improve logic sharing, I think that work can proceed in parallel.
Cheers,
Dmitri.
On Tue, Sep 22, 2026 at 5:55 PM EJ Wang <ej...@gmail.com>
wrote:
> Thanks for the recap, Dmitri. I'd like to follow up on the three points I
> raised during the sync. We didn't have much time to discuss them, so it
> would help to confirm these before choosing an SPI design.
>
> 1. Which backend assumptions should we design for?
>
> Within one Polaris server instance, do we assume entities, grants and
> secrets all use the same database backend, or must we support mixing
> backends across them?
> Must every supported persistence implementation provide transactional
> guarantees? Does the database itself need to provide transactions, or can
> Polaris build that capability on top of simpler database operations?
>
> I'm happy to work with a homogeneous assumption if that's the intended
> scope. Existing NoSQL illustrates the second question: it can publish
> changes to multiple objects under one reference using a single conditional
> pointer update.
>
> 2. How do we protect the data used by validation?
>
> Your storage-configuration and location-overlap examples make the problem
> concrete. Consider these races:
>
> A table creation passes validation against the catalog's allowed locations,
> then another request changes that configuration before the table is
> committed.
> Two table-creation requests each check for overlapping locations, both see
> no overlap, and then create tables at conflicting locations.
>
> Checking only the record being written cannot cover these dependencies. The
> first also depends on the catalog configuration. The second depends on the
> absence of another overlapping table.
>
> Could the persistence layer protect these reads together with the writes,
> using backend-internal versions/tokens or database isolation? Shared code
> would still need to identify the required reads and checks. Could we keep
> the mechanism for protecting them inside each backend?
>
> 3. Business rules are duplicated across backend implementations
>
> Today, the TreeMap
> <
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481
> >
> and JDBC
> <
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046
> >
> implementations each load a secret, check its principal, apply the
> rotation/reset steps and write it back.
>
> Changing these business rules therefore requires keeping multiple
> implementations in sync. A fix applied to one backend can be missed in
> another, causing the same Polaris operation to behave differently depending
> on the database.
>
> My proposal is to implement each business workflow once, against a common
> set of storage operations with defined consistency and failure semantics.
> The shared workflow should not need to know whether those operations are
> implemented through JDBC transactions, NoSQL reference CAS or another
> mechanism. Each backend would implement the same storage contract, keeping
> its execution details internal. A new database backend could then reuse the
> existing business workflows.
>
> Do these seem like reasonable starting points? We can then evaluate the SPI
> against the original failure cases on both JDBC and existing NoSQL.
>
> -ej
>
> On Fri, Sep 18, 2026 at 12:10 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Side note: I just came across [5541], which shows a case where a failed
> > request leaves persisted side effects.
> >
> > This can be seen as a coding mistake, of course. However, this is about
> > executing one request as a single, atomic unit. This is something that
> > probably needs foundational support and coding patterns in Polaris code.
> >
> > Cheers,
> > Dmitri.
> > [5541] https://github.com/apache/polaris/pull/5541
> >
> > On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi All,
> > >
> > > Here's a recap of the Community Sync discussion as I remember it. I'm
> > sure
> > > my recollection is not complete, so please add your thoughts to this
> > thread.
> > >
> > > * Polaris traditionally relies on entity version number to ensure
> > > consistency of changes.
> > >
> > > This works well for internal RBAC grants, which also have version
> numbers
> > > cross-referenced at the Persistence layer.
> > >
> > > I do not think this works in more general cases like validating
> locations
> > > wrt catalog-level storage configuration.
> > >
> > > Another difficult case involves location overlaps between concurrent
> > table
> > > creations.
> > >
> > > * We talked about how Persistence calls related to RDBMS transactions
> (in
> > > the JDBC persistence case).
> > >
> > > There were some concerns raised about whether we need to bind
> Persistence
> > > API to transactions explicitly.
> > >
> > > I think it is a valid concern. At the same time, the Persistence API
> > > should be clear about consistency guarantees across all backend
> > > implementations. I think this needs more attention now that the matter
> of
> > > milti-entioty changes became prominent in [5035].
> > >
> > > I personally believe that RDBMS-based Persistence in Polaris has to use
> > > Serializable transaction isolation. Consequently, the core code needs
> to
> > > provide mechanisms to allow the Persistence implementation to properly
> > > connect reads and writes from API requests to JDBC transactions...
> > whether
> > > we use "transation" as an explicit terms in the SPI or not.
> > >
> > > Note that currently, the same API request performs reads and writes in
> > > _separate_ RDBMS transactions (even separate connections).
> > >
> > > * We did not discuss Persistence SPI implications in the call, but
> > > currently each Persistence call is assumed to be one distinct and
> atomic
> > > change (please correct me if I'm wrong).
> > >
> > > Consequently, supporting new use cases involves introducing new SPI
> > > methods with many parameters. This can be seen in PR [5035]. This tends
> > to
> > > overcomplicate the SPI.
> > >
> > > Ideally, I think the SPI should be simplified to deal with multi-entity
> > > and single-entity changes in a coherent manner to avoid any ambiguity
> > about
> > > which method the caller should use.
> > >
> > > [5035] https://github.com/apache/polaris/pull/5035
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > wrote:
> > >
> > >> Heads up: This discussion is on the Community Sync call agenda for
> Sept
> > >> 17.
> > >>
> > >> Interested parties, please plan to attend, if possible :)
> > >>
> > >> Cheers,
> > >> Dmitri.
> > >>
> > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <di...@apache.org>
> > >> wrote:
> > >>
> > >>> Hi JB,
> > >>>
> > >>> I've added an agenda item to the next community sync call for this
> > (Sept
> > >>> 17, 2026).
> > >>>
> > >>> Re: key points: did you mean a real GH discussion or dev email (this
> > >>> thread)? Just double checking :) I'm fine with either approach.
> > >>>
> > >>> Cheers,
> > >>> Dmitri.
> > >>>
> > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > >>> wrote:
> > >>>
> > >>>> Hi Dmitri,
> > >>>>
> > >>>> I agree on the need to align on these points, though I'm not
> entirely
> > >>>> sure a dedicated meeting is necessary. Let's start by using some
> time
> > >>>> during the next community meeting to discuss it.
> > >>>>
> > >>>> In the meantime, what if we outline the key points in a GitHub
> > >>>> Discussion first? We can then use that as a reference during the
> > >>>> meeting.
> > >>>>
> > >>>> Regards,
> > >>>> JB
> > >>>>
> > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <
> dimas@apache.org
> > >
> > >>>> wrote:
> > >>>> >
> > >>>> > Hi All,
> > >>>> >
> > >>>> > My impression from this thread is that it might be time for a call
> > to
> > >>>> > discuss all the related issues and try to arrive at a shared
> > >>>> implementation
> > >>>> > plan.
> > >>>> >
> > >>>> > I know meetings are not ideal, but in this case it might be
> > >>>> beneficial as a
> > >>>> > means for achieving a common understanding of the set of problems
> > and
> > >>>> > priorities related to this thread. A dedicated metrics meeting
> > worked
> > >>>> well
> > >>>> > from my POV.
> > >>>> >
> > >>>> > Allocating a time slot in the next community meeting might be an
> > >>>> option,
> > >>>> > although I think we might need the full hour in this case.
> > >>>> >
> > >>>> > WDYT?
> > >>>> >
> > >>>> > Thanks,
> > >>>> > Dmitri.
> > >>>> >
> > >>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <
> > >>>> prithvisivasankar@gmail.com>
> > >>>> > wrote:
> > >>>> >
> > >>>> > > Hi all,
> > >>>> > >
> > >>>> > > I rewrote https://github.com/apache/polaris/pull/5035,
> following
> > >>>> the
> > >>>> > > discussion to focus only on the SPI foundation.
> > >>>> > >
> > >>>> > > What is in the PR now:
> > >>>> > >
> > >>>> > > - A written manager-level consistency contract in
> > >>>> > > site/content/in-dev/unreleased/persistence-consistency.md.
> > >>>> > > - CAS-aware EntityMutation / GrantMutation records.
> > >>>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet,
> with
> > >>>> backends
> > >>>> > > opting in via supportsAtomicMixedCommit().
> > >>>> > > - commitChangeSet implementations for the in-memory TreeMap
> > >>>> backend and
> > >>>> > > JDBC.
> > >>>> > > - AtomicOperationMetaStoreManager and
> > >>>> TransactionalMetaStoreManagerImpl
> > >>>> > > grant/revoke paths now build one change set per operation,
> > >>>> falling back
> > >>>> > > to
> > >>>> > > individual operations when the backend does not support mixed
> > >>>> commits.
> > >>>> > > - Tests for the change-set API and TreeMap atomic commits.
> > >>>> > >
> > >>>> > > I intentionally left out:
> > >>>> > >
> > >>>> > > - Entity deletes in MetaStoreChangeSet.
> > >>>> > > - Refactoring createCatalog, dropEntity, and renameEntity to
> > use
> > >>>> change
> > >>>> > > sets.
> > >>>> > > - NoSQL support for commitChangeSet.
> > >>>> > >
> > >>>> > > I am treating this as Phase 1 so the contract and the SPI
> > primitive
> > >>>> can be
> > >>>> > > reviewed before the larger operation migrations land.
> > >>>> > >
> > >>>> > > Could you please take a look? In particular, I would like to
> know
> > >>>> whether
> > >>>> > > the contract captures the consensus so far, or if it needs to
> > >>>> address retry
> > >>>> > > / external-work coordination before we merge this foundation.
> > >>>> > >
> > >>>> > > Thanks,
> > >>>> > > Prithvi S
> > >>>> > >
> > >>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <
> > >>>> dimas@apache.org>
> > >>>> > > wrote:
> > >>>> > >
> > >>>> > > > Hi Robert,
> > >>>> > > >
> > >>>> > > > I agree that the client-visible operation (e.g. REST API
> > request)
> > >>>> > > consists
> > >>>> > > > of many distinct phases. The database / persistence changes
> are
> > >>>> just one
> > >>>> > > of
> > >>>> > > > these phases. STS (as an example) is another. My point about
> > >>>> transactions
> > >>>> > > > in JDBC Persistence applies to the former (database changes)
> > >>>> phase - one
> > >>>> > > > retry attempt there should ideally involve exactly one
> > >>>> transaction.
> > >>>> > > >
> > >>>> > > > We certainly need logic inside Polaris Servers to to
> coordinate
> > >>>> requests
> > >>>> > > to
> > >>>> > > > various external systems depending on outcomes from previous
> > >>>> request
> > >>>> > > > processing phases.
> > >>>> > > >
> > >>>> > > > I also agree that validation based on database state should be
> > >>>> repeated
> > >>>> > > > from scratch if we retry the database changes.
> > >>>> > > >
> > >>>> > > > However, internal RBAC authorization naturally has to happen
> > >>>> inside the
> > >>>> > > > database update phase (read-check-write). IIRC, that is part
> of
> > >>>> the
> > >>>> > > > implicit consistency guarantees, which were discussed during
> > early
> > >>>> > > project
> > >>>> > > > months (but never got written down in full clarity,
> > >>>> unfortunately). If
> > >>>> > > the
> > >>>> > > > Authorization check is inside that phase for internal RBAC, I
> > >>>> suppose it
> > >>>> > > > will be there for other Authorizers too and that can involve
> > >>>> external
> > >>>> > > > systems (e.g. OPA / Ranger). We can certainly expect
> > >>>> authorization to be
> > >>>> > > > efficient, but I'm not sure how quick it can be in practice.
> > >>>> There's
> > >>>> > > > certainly potential for delays in some (perhaps infrequent)
> > cases.
> > >>>> > > >
> > >>>> > > > Cheers,
> > >>>> > > > Dmitri.
> > >>>> > > >
> > >>>> > > >
> > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <sn...@snazy.de>
> > >>>> wrote:
> > >>>> > > >
> > >>>> > > > > Hi Dmitri,
> > >>>> > > > >
> > >>>> > > > > I agree that one logical operation needs a consistent view
> of
> > >>>> the whole
> > >>>> > > > > backend state.
> > >>>> > > > > What worries me is treating one long JDBC transaction as the
> > >>>> boundary
> > >>>> > > of
> > >>>> > > > > that operation and then retrying the complete request if the
> > >>>> commit
> > >>>> > > > fails.
> > >>>> > > > >
> > >>>> > > > > The request can also write metadata to object storage or
> call
> > >>>> STS.
> > >>>> > > > > Each system can leave us with an outcome whose certainty we
> > >>>> cannot
> > >>>> > > know.
> > >>>> > > > > The database may have committed before the connection was
> > lost,
> > >>>> an
> > >>>> > > > > object-store write may have succeeded before a timeout, or
> STS
> > >>>> may have
> > >>>> > > > > issued credentials before its response was lost.
> > >>>> > > > > A database rollback cannot undo any of those effects, and
> > >>>> retrying the
> > >>>> > > > > whole request may repeat them.
> > >>>> > > > >
> > >>>> > > > > So I think the contract has to separate the overall
> operation
> > >>>> from each
> > >>>> > > > > attempt to commit it.
> > >>>> > > > > Each database or backend commit attempt should stay short.
> > >>>> > > > > When we retry, validation and authorization need to use the
> > >>>> state for
> > >>>> > > > that
> > >>>> > > > > new attempt, and we need clear rules for which external work
> > >>>> can be
> > >>>> > > > reused,
> > >>>> > > > > repeated, cleaned up, or reconciled after an uncertain
> > outcome.
> > >>>> > > > >
> > >>>> > > > > Some contextual state could be useful for carrying that
> state.
> > >>>> > > > > But the JDBC connection and transaction should be an
> > >>>> implementation
> > >>>> > > > detail
> > >>>> > > > > inside it, not the boundary of the complete operation.
> > >>>> > > > >
> > >>>> > > > > Cheers,
> > >>>> > > > > Robert
> > >>>> > > > >
> > >>>> > > > >
> > >>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <
> > >>>> dimas@apache.org
> > >>>> > > >
> > >>>> > > > > wrote:
> > >>>> > > > >
> > >>>> > > > > > Hi All,
> > >>>> > > > > >
> > >>>> > > > > > The idea of a backend-agnostic change-set primitive sounds
> > >>>> good.
> > >>>> > > > > >
> > >>>> > > > > > However, I am not sure it is sufficient for all the use
> > cases
> > >>>> we
> > >>>> > > > touched
> > >>>> > > > > > here. More specifically, the state read by validation code
> > is
> > >>>> not
> > >>>> > > > > > necessarily reflected in the change set:
> > >>>> > > > > >
> > >>>> > > > > > * Unchanged entities might be considered by validation,
> but
> > >>>> changed
> > >>>> > > in
> > >>>> > > > a
> > >>>> > > > > > parallel request.
> > >>>> > > > > > * Even for changed entities the state read by validation
> may
> > >>>> differ
> > >>>> > > > from
> > >>>> > > > > > the state being altered. Some validation code talks to the
> > >>>> MetaStore
> > >>>> > > > > > directly, outside of the data produced by the Resolver.
> > >>>> > > > > >
> > >>>> > > > > > I do not think Polaris offers any explicit mechanisms
> (ATM)
> > >>>> to ensure
> > >>>> > > > > > consistency between these reads and subsequent writes.
> > >>>> > > > > >
> > >>>> > > > > > As far as JDBC goes, running an overarching Transaction
> > >>>> across all
> > >>>> > > > reads
> > >>>> > > > > > and writes in the same request, at the SERIALIZABLE
> > isolation
> > >>>> level
> > >>>> > > in
> > >>>> > > > > the
> > >>>> > > > > > backing RDBMS could solve the problem, I think.
> > >>>> > > > > >
> > >>>> > > > > > Indeed, such a transaction might be long to accommodate
> > calls
> > >>>> made by
> > >>>> > > > > > Polaris to external storage, etc. However, is that a
> > problem?
> > >>>> I think
> > >>>> > > > the
> > >>>> > > > > > alternative is for the JDBC Persistence impl. to
> "manually"
> > >>>> track all
> > >>>> > > > > reads
> > >>>> > > > > > under the same request and redo them in the small
> > transaction
> > >>>> that
> > >>>> > > > > persists
> > >>>> > > > > > the writes. This will also add RDBMS overhead and require
> > >>>> complex
> > >>>> > > code
> > >>>> > > > in
> > >>>> > > > > > Polaris to handle the data properly.
> > >>>> > > > > >
> > >>>> > > > > > Tracking the request-wide transaction can be done only for
> > >>>> JDBC
> > >>>> > > without
> > >>>> > > > > > leaking "transaction" concepts to the NoSQL Persistence, I
> > >>>> think.
> > >>>> > > NoSQL
> > >>>> > > > > > will use other mechanisms to ensure read/write
> consistency.
> > >>>> > > > > >
> > >>>> > > > > > Connecting to Robert's email (a parallel branch in this
> > >>>> discussion),
> > >>>> > > > I'd
> > >>>> > > > > > like to propose this approach:
> > >>>> > > > > >
> > >>>> > > > > > * Each request establishes a "Data Context"
> > >>>> > > > > > - In JDBC the Data Context corresponds to a JDBC
> > Connection
> > >>>> + Tx
> > >>>> > > > > > - In NoSQL the Data Context tracks one or more reference
> > >>>> hashes
> > >>>> > > > > > * All Persistence access in the same request goes through
> > the
> > >>>> same
> > >>>> > > Data
> > >>>> > > > > > Context
> > >>>> > > > > > * All Persistence changes are committed once at the end of
> > the
> > >>>> > > request
> > >>>> > > > > > - Not all changes have to be globally atomic. For
> example,
> > >>>> changes
> > >>>> > > in
> > >>>> > > > > > Catalogs A and B do not have to be atomic with respect to
> > >>>> each other.
> > >>>> > > > We
> > >>>> > > > > > can go deeper into this later. This is relevant to NoSQL.
> > >>>> > > > > > - Transactional backends like JDBC can, of course,
> choose
> > >>>> to make
> > >>>> > > all
> > >>>> > > > > > changes globally atomic.
> > >>>> > > > > > * A commit can fail in two main ways:
> > >>>> > > > > > - A retriable failure like an optimistic lock error or
> Tx
> > >>>> > > > > serializability
> > >>>> > > > > > error
> > >>>> > > > > > - A non-triable logical error (e.g. entity not found)
> > >>>> > > > > > * On a retriable error the whole request is re-attempted
> (as
> > >>>> if
> > >>>> > > > > resubmitted
> > >>>> > > > > > by a client) a few times (configurable timeout).
> > >>>> > > > > >
> > >>>> > > > > > WDYT?
> > >>>> > > > > >
> > >>>> > > > > > Cheers,
> > >>>> > > > > > Dmitri.
> > >>>> > > > > >
> > >>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré <
> > >>>> > > jb@nanthrax.net
> > >>>> > > > >
> > >>>> > > > > > wrote:
> > >>>> > > > > >
> > >>>> > > > > > > Hi all
> > >>>> > > > > > >
> > >>>> > > > > > > I think Robert has a good point.
> > >>>> > > > > > >
> > >>>> > > > > > > If the atomicity guarantee lives only in
> BasePersistence,
> > >>>> then the
> > >>>> > > > > > > manager contract can't tell a caller whether the state
> it
> > >>>> read for
> > >>>> > > > > > > validation/authorization/credential-vending is the same
> > >>>> state that
> > >>>> > > > > > > eventually commits. That's the actual bug class behind
> the
> > >>>> JDBC
> > >>>> > > > > > > symptoms (not any single operation being non-atomic, but
> > the
> > >>>> > > contract
> > >>>> > > > > > > being silent about it. Every operation-specific fix
> (like
> > >>>> #4939,
> > >>>> > > > #5035
> > >>>> > > > > > > or #5095) re-answers this question locally and it keeps
> > >>>> recurring.
> > >>>> > > So
> > >>>> > > > > > > the deliverable should start with a written consistency
> > >>>> contract at
> > >>>> > > > > > > the manager level.
> > >>>> > > > > > >
> > >>>> > > > > > > I'm not sure migrating everyone to
> > >>>> > > TransactionalMetaStoreManagerImpl
> > >>>> > > > > > > is a good idea. It would tie the logical change set to a
> > DB
> > >>>> > > > > > > transaction spanning the REST request. It means:
> > >>>> > > > > > > - it holds a durable transaction open across slow
> external
> > >>>> work
> > >>>> > > > > > > (credential vending, OPA, Ranger, ...)
> > >>>> > > > > > > - it doesn't map to NoSQL
> > >>>> > > > > > > - It wraps single-row updates in runWiithinTransaction,
> > >>>> which is a
> > >>>> > > > > > overhead
> > >>>> > > > > > >
> > >>>> > > > > > > So, I think the transactional manager isn't a portable
> > >>>> target. It's
> > >>>> > > > > > > "only" one backend's strategy.
> > >>>> > > > > > >
> > >>>> > > > > > > I think Privthi's approach is right. A backend-agnostic
> > >>>> change-set
> > >>>> > > > > > > primitive with a documented fallback is the correct
> shape.
> > >>>> There is
> > >>>> > > > > > > one caveat: without carrying the original entitiy (not
> > just
> > >>>> the new
> > >>>> > > > > > > one) the change set can't express optimistic
> concurrency.
> > >>>> > > > > > >
> > >>>> > > > > > > I propose the following multi-steps approach:
> > >>>> > > > > > > 1. We write the manager-level consistency contract as
> > >>>> Robert asked.
> > >>>> > > > It
> > >>>> > > > > > > should include the explicit statement that a logical
> > change
> > >>>> set is
> > >>>> > > > not
> > >>>> > > > > > > a request-scope DB transaction.
> > >>>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency
> > pattern)
> > >>>> > > baseline
> > >>>> > > > > > > first-class in EntityMutation before merging the SPI
> > >>>> > > > > > > 3. Then we refactor
> createCatalog/dropEntity/renameEntity.
> > >>>> > > > > > >
> > >>>> > > > > > > Thoughts?
> > >>>> > > > > > >
> > >>>> > > > > > > Regards
> > >>>> > > > > > > JB
> > >>>> > > > > > >
> > >>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <
> > >>>> snazy@snazy.de>
> > >>>> > > wrote:
> > >>>> > > > > > > >
> > >>>> > > > > > > > Hi all,
> > >>>> > > > > > > >
> > >>>> > > > > > > > Yufei's clarification about where the atomicity
> > guarantee
> > >>>> is
> > >>>> > > > defined
> > >>>> > > > > > > seems
> > >>>> > > > > > > > important.
> > >>>> > > > > > > > If it is a property of the lower-level BasePersistence
> > >>>> contract
> > >>>> > > > > rather
> > >>>> > > > > > > than
> > >>>> > > > > > > > the general PolarisMetaStoreManager contract, the
> > general
> > >>>> > > contract
> > >>>> > > > > does
> > >>>> > > > > > > not
> > >>>> > > > > > > > tell callers whether the state used for validation,
> > >>>> > > authorization,
> > >>>> > > > or
> > >>>> > > > > > > > credential vending is consistent with the change that
> > >>>> eventually
> > >>>> > > > > > commits.
> > >>>> > > > > > > >
> > >>>> > > > > > > > The current PRs suggest that operation-specific
> > >>>> multi-object
> > >>>> > > > methods
> > >>>> > > > > > can
> > >>>> > > > > > > > fix individual cases, while leaving the same contract
> > >>>> question to
> > >>>> > > > > recur
> > >>>> > > > > > > for
> > >>>> > > > > > > > each new case.
> > >>>> > > > > > > >
> > >>>> > > > > > > > I would also avoid defining a logical change set as a
> > >>>> database
> > >>>> > > > > > > transaction
> > >>>> > > > > > > > around an entire REST request.
> > >>>> > > > > > > > That would tie the contract to the backend and
> database,
> > >>>> and
> > >>>> > > could
> > >>>> > > > > keep
> > >>>> > > > > > > the
> > >>>> > > > > > > > durable attempt open across slow external work.
> > >>>> > > > > > > >
> > >>>> > > > > > > > So is the choice really between the two current
> manager
> > >>>> > > > > > implementations,
> > >>>> > > > > > > or
> > >>>> > > > > > > > do we first need to revisit the boundary and
> guarantees
> > >>>> exposed
> > >>>> > > to
> > >>>> > > > > > their
> > >>>> > > > > > > > callers?
> > >>>> > > > > > > >
> > >>>> > > > > > > > Cheers,
> > >>>> > > > > > > > Robert
> > >>>> > > > > > > >
> > >>>> > > > > > > >
> > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov <
> > >>>> > > > > dimas@apache.org
> > >>>> > > > > > >
> > >>>> > > > > > > > wrote:
> > >>>> > > > > > > >
> > >>>> > > > > > > > > Hi Yufei,
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > Thanks for the link. I stand corrected. The "one
> > atomic
> > >>>> change
> > >>>> > > > per
> > >>>> > > > > > > method"
> > >>>> > > > > > > > > contract as defined in javadoc does apply only to
> > >>>> > > BasePersistence
> > >>>> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates
> > to
> > >>>> > > > > > > BasePersistence).
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > Note that all non-test Persistence implementations
> in
> > >>>> the
> > >>>> > > Polaris
> > >>>> > > > > > > codebase
> > >>>> > > > > > > > > extend those classes (which is probably why I was
> > >>>> confused
> > >>>> > > about
> > >>>> > > > > > > atomicity
> > >>>> > > > > > > > > expectations).
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > However, this creates a gap in the Persistence SPI
> > >>>> > > specification.
> > >>>> > > > > If
> > >>>> > > > > > > > > other PolarisMetaStoreManager implementations do not
> > >>>> have to
> > >>>> > > > comply
> > >>>> > > > > > > with
> > >>>> > > > > > > > > this principle, it will create a conceptual
> difficulty
> > >>>> at call
> > >>>> > > > > sites.
> > >>>> > > > > > > How
> > >>>> > > > > > > > > can PolarisMetaStoreManager callers reason about
> > >>>> consistency
> > >>>> > > and
> > >>>> > > > > > > durability
> > >>>> > > > > > > > > behaviours in general?
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > I believe we need to address that as part of this
> > >>>> discussion.
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > Cheers,
> > >>>> > > > > > > > > Dmitri.
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <
> > >>>> flyrain000@gmail.com
> > >>>> > > >
> > >>>> > > > > > wrote:
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with the
> > >>>> idea that
> > >>>> > > one
> > >>>> > > > > > > method
> > >>>> > > > > > > > > call
> > >>>> > > > > > > > > > means one atomic change.
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > If the "MetaStore SPI" refers to the interface
> > >>>> > > > > > > PolarisMetaStoreManager, I
> > >>>> > > > > > > > > > don't think we've ever state each method to be
> > >>>> atomic. We did
> > >>>> > > > > > clarify
> > >>>> > > > > > > > > > atomicity[1] in the interface BasePersistence
> > though.
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > 1.
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > >
> > >>>> > > > > >
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>>
> >
> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > Yufei
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri
> > Bourlatchkov <
> > >>>> > > > > > > dimas@apache.org>
> > >>>> > > > > > > > > > wrote:
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > > Hi Yufei,
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > I agree that JDBC transactions must be handled
> > more
> > >>>> > > > explicitly.
> > >>>> > > > > > > > > However,
> > >>>> > > > > > > > > > > I'm not sure that simply moving to
> > >>>> > > > > > > TransactionalMetaStoreManagerImpl is
> > >>>> > > > > > > > > > > sufficient.
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with the
> > >>>> idea that
> > >>>> > > one
> > >>>> > > > > > > method
> > >>>> > > > > > > > > call
> > >>>> > > > > > > > > > > means one atomic change [1]. The "transactional"
> > >>>> MetaStore
> > >>>> > > > > impl.
> > >>>> > > > > > is
> > >>>> > > > > > > > > but a
> > >>>> > > > > > > > > > > sub-case of that. It cannot alter the high-level
> > >>>> contract.
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > We could add SPI methods having multiple object
> > >>>> parameters
> > >>>> > > to
> > >>>> > > > > > > represent
> > >>>> > > > > > > > > > > grouped changes, but I am not sure it will be a
> > >>>> sound
> > >>>> > > design.
> > >>>> > > > > > This
> > >>>> > > > > > > will
> > >>>> > > > > > > > > > > bloat the interface surfaces and require extra
> > >>>> impl. effort
> > >>>> > > > for
> > >>>> > > > > > > each
> > >>>> > > > > > > > > > > backend type. More importantly, adding multi-arg
> > >>>> change
> > >>>> > > > methods
> > >>>> > > > > > > still
> > >>>> > > > > > > > > > won't
> > >>>> > > > > > > > > > > address the problem of reads being consistent
> with
> > >>>> writes,
> > >>>> > > > > > because
> > >>>> > > > > > > each
> > >>>> > > > > > > > > > > method call will still be independent regarding
> > the
> > >>>> data
> > >>>> > > > stored
> > >>>> > > > > > in
> > >>>> > > > > > > the
> > >>>> > > > > > > > > > > database.
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > I tend to think we need to introduce a "change
> > set"
> > >>>> or
> > >>>> > > > "atomic
> > >>>> > > > > > > batch"
> > >>>> > > > > > > > > > > concept to core Persistence and associate each
> > REST
> > >>>> API
> > >>>> > > > request
> > >>>> > > > > > > with
> > >>>> > > > > > > > > one
> > >>>> > > > > > > > > > > such change set, which will be committed (or
> > rolled
> > >>>> back)
> > >>>> > > at
> > >>>> > > > > the
> > >>>> > > > > > > end of
> > >>>> > > > > > > > > > the
> > >>>> > > > > > > > > > > request. I believe Ayush mentioned a similar
> > >>>> concept in PR
> > >>>> > > > 4939
> > >>>> > > > > > > [2]. In
> > >>>> > > > > > > > > > > JDBC each change set will naturally be
> associated
> > >>>> with an
> > >>>> > > > RDBMS
> > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each atomic
> > >>>> change set
> > >>>> > > > will
> > >>>> > > > > be
> > >>>> > > > > > > > > > > associated with one CAS operation on the
> > underlying
> > >>>> > > database.
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > [1]
> > >>>> > > > > > >
> > >>>> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > [2]
> > >>>> > > > > > >
> > >>>> https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > Cheers,
> > >>>> > > > > > > > > > > Dmitri.
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
> > >>>> > > > flyrain000@gmail.com
> > >>>> > > > > >
> > >>>> > > > > > > wrote:
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > > Thanks for raising this, Dmitri.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > These are valid concerns, and they were
> already
> > >>>> > > recognized
> > >>>> > > > > when
> > >>>> > > > > > > we
> > >>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At
> that
> > >>>> time, we
> > >>>> > > > > chose
> > >>>> > > > > > > to use
> > >>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC
> due
> > >>>> to the
> > >>>> > > > > > > simplicity. I
> > >>>> > > > > > > > > > > think
> > >>>> > > > > > > > > > > > most of the issues mentioned here can already
> be
> > >>>> > > addressed
> > >>>> > > > by
> > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > For example, rename is already wrapped in a
> > >>>> transaction
> > >>>> > > in
> > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1].
> > Similarly,
> > >>>> catalog
> > >>>> > > > > > > creation,
> > >>>> > > > > > > > > > which
> > >>>> > > > > > > > > > > > involves reading and creating multiple
> objects,
> > >>>> is also
> > >>>> > > > > > executed
> > >>>> > > > > > > > > > within a
> > >>>> > > > > > > > > > > > transaction [2].
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > I see two possible directions:
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > 1.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager
> > >>>> together with
> > >>>> > > the
> > >>>> > > > > > > > > persistence
> > >>>> > > > > > > > > > > > backends (such as JDBC) to provide the
> > required
> > >>>> > > > > consistency
> > >>>> > > > > > > > > > guarantees
> > >>>> > > > > > > > > > > > for
> > >>>> > > > > > > > > > > > specific operations, similar to what
> > >>>> > > > > > > > > > TransactionalMetaStoreManagerImpl
> > >>>> > > > > > > > > > > > does.
> > >>>> > > > > > > > > > > > 2.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > Migrate the persistence backends (such as
> > >>>> JDBC) to use
> > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl directly.
> > We
> > >>>> may
> > >>>> > > have
> > >>>> > > > to
> > >>>> > > > > > > deal
> > >>>> > > > > > > > > with
> > >>>> > > > > > > > > > > > transactional semantic mismatches across
> > >>>> different
> > >>>> > > > > > persistence
> > >>>> > > > > > > > > > > backends.
> > >>>> > > > > > > > > > > > For example, we would likely avoid using
> > JDBC's
> > >>>> > > > > > > > > > `runWithinTransaction`
> > >>>> > > > > > > > > > > > for
> > >>>> > > > > > > > > > > > single row updates, which adds additional
> > >>>> overhead and
> > >>>> > > > > > > complexity
> > >>>> > > > > > > > > > > > without
> > >>>> > > > > > > > > > > > benefits.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > References:
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > 1.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > >
> > >>>> > > > > >
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>>
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > >>>> > > > > > > > > > > > 2.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > >
> > >>>> > > > > >
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>>
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > Yufei
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri
> > >>>> Bourlatchkov <
> > >>>> > > > > > > > > dimas@apache.org>
> > >>>> > > > > > > > > > > > wrote:
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > Hi all,
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a
> > couple
> > >>>> of
> > >>>> > > > > > interesting
> > >>>> > > > > > > PRs:
> > >>>> > > > > > > > > > > > > [4939], [5035].
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > It looks like people are starting to
> encounter
> > >>>> > > > consistency
> > >>>> > > > > > > issues
> > >>>> > > > > > > > > in
> > >>>> > > > > > > > > > > > > JDBC persistence.
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > The PRs provide valuable insight into the
> > >>>> underlying
> > >>>> > > > > issues.
> > >>>> > > > > > > They
> > >>>> > > > > > > > > > offer
> > >>>> > > > > > > > > > > > > incremental fixes that can work. However, I
> > >>>> believe it
> > >>>> > > is
> > >>>> > > > > > time
> > >>>> > > > > > > for
> > >>>> > > > > > > > > > the
> > >>>> > > > > > > > > > > > > Polaris community to review and improve this
> > >>>> area of
> > >>>> > > the
> > >>>> > > > > > > codebase
> > >>>> > > > > > > > > > > > > holistically.
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > By this, I mean finding a solution that can
> be
> > >>>> applied
> > >>>> > > to
> > >>>> > > > > all
> > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC,
> NoSQL)
> > >>>> and
> > >>>> > > > addresses
> > >>>> > > > > > > these
> > >>>> > > > > > > > > > > > > aspects:
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent
> changes
> > >>>> where
> > >>>> > > the
> > >>>> > > > > > > service
> > >>>> > > > > > > > > > reads
> > >>>> > > > > > > > > > > > > and validates current catalog state, then
> > >>>> commits a
> > >>>> > > > > change
> > >>>> > > > > > > (e.g.
> > >>>> > > > > > > > > > > > > name clashes during renames).
> > >>>> > > > > > > > > > > > > * Supporting consistent but independent
> > changes
> > >>>> to RBAC
> > >>>> > > > > > grants
> > >>>> > > > > > > and
> > >>>> > > > > > > > > > > > > MetaStore entities. This independence is
> > >>>> needed to
> > >>>> > > > > support
> > >>>> > > > > > > > > > > > > external authorizers like OPA and Ranger.
> > >>>> > > > > > > > > > > > > * Supporting atomic changes across multiple
> > >>>> similar
> > >>>> > > > > entities.
> > >>>> > > > > > > > > > > > > * Supporting authorization-based filtering
> of
> > >>>> list
> > >>>> > > > > operations
> > >>>> > > > > > > (cf.
> > >>>> > > > > > > > > > > > > [4831]).
> > >>>> > > > > > > > > > > > > * Supporting credential-vending decisions
> that
> > >>>> are
> > >>>> > > rooted
> > >>>> > > > > in
> > >>>> > > > > > > the
> > >>>> > > > > > > > > > > > > exact state of the catalog.
> > >>>> > > > > > > > > > > > > * Supporting server-side retries for
> transient
> > >>>> > > > persistence
> > >>>> > > > > > > failures
> > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures).
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > Please share your comments and ideas.
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > [4831]
> > >>>> https://github.com/apache/polaris/pull/4831
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > [4939]
> > >>>> https://github.com/apache/polaris/pull/4939
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > [5035]
> > >>>> https://github.com/apache/polaris/pull/5035
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > Thanks,
> > >>>> > > > > > > > > > > > > Dmitri
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > >
> > >>>> > > > > >
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>>
> > >>>
> >
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Dennis,
This is not a reply to your email, but a quick note
regarding PolarisResolutionManifest.
From my POV, the most fundamental divergence from the idea it represents is
the "optimized" location check, which goes directly to persistence and does
not work with the resolver or manifest on the request side. That is, it
takes one entity on input and finds conflicts with other entities via
low-level persistence operations.
Cheers,
Dmitri.
On Wed, Sep 30, 2026 at 8:42 PM Dennis Huo <dh...@apache.org> wrote:
> Agree we should formulate in terms of Persistence SPIs first.
>
> IIRC we talked about this in one of the community syncs but I forgot to
> bring it here -- the PolarisResolutionManifest in concept already
> represents *exactly* the set of "which elements were engaged during request
> processing".
>
> Adherence may have drifted, but at least early on, we made sure all the
> IcebergCatalog logic did *not* get to do new direct reads from the
> persistence layer, but instead needs to "read" from the *authorized* set of
> entities contained in a PolarisResolutionManifest (or declared as a
> "passthroughPath" == allow reading fresh from DB but had to pre-register it
> for authz purposes).
>
> It's important to keep this unified anyways because it really serves two
> things:
>
> 1. Declares the set of entities whose state you expect to resolve/snapshot
> up-front in the operation-handling, that the IcebergCatalog logic then
> operates on later; also explicitly declare which ones require fetching
> *fresh* versions mid-operation (addPassthroughPath)
> 2. Enforces that RBAC authz is performed against that pre-declared set of
> entities -- if later processign were allowed to access arbitrary other
> entities to make handler decisions, then how did we "prove" that the
> authorization engine actually allowed the request to access those arbitrary
> other entities? Even if it's just for internal-processing purposes, the
> very fact that something got accessed means there's a route to accidental
> leakage of unauthorized data.
>
> So in a way, solid/provable authz and solid/provable consistency go
> hand-in-hand.
>
> I think PolarisResolutionManifest in its current form is probably a bit
> opaque and doesn't 100% convey the semantics we want quite yet, but it
> seems like the right SPI starting point to extend or at least evolve from.
> Basically, aside from addPath and addPassthroughPath, there could be a
> notion of annotating a given path as being one of the "dependent states" to
> carry over into the later commit. Conceptually:
>
> // Prepare operation processing
> resolutionManifest.prefetch(rootCatalog, mustBeUnchangedAtCommitTime =
> false); // For this operation let's pretend we don't care if the root
> catalog changes before we commit the change to the tables
> resolutionManifest.prefetch(parentNamespace, mustBeUnchangedAtCommitTime =
> false); // For this operation let's pretend we don't care if the parent
> namespace changes before we commit the change to the tables
> resolutionManifest.prefetch(table1, mustBeUnchangedAtCommitTime = true);
> // Assume multi-table commit - table1 will have changes and also must not
> have already changed at commit time
> resolutionManifest.prefetch(table2, mustBeUnchangedAtCommitTime = true);
> // Same for table2
>
> // .... Do all the Iceberg processing work
>
> // When we commit, we use the resolutionManifest as the ledger of things we
> used that must still be at their unchanged version at time of commit
> metastore.doConditionalBatchOmmit(List.of(table1Updated, table2Updated),
> resolutionManifest.getSetOfUnchangedEntityRequirements());
>
> Overall this fits the optimistic-concurrency theme of Iceberg and Polaris
> in general, and would *intentionally* not try to express a full-fledged
> "Transaction" session in which you do a bunch of reads/writes/reads/writes
> on a single entity as if you're operating in an uncommitted block on a
> RDBMS, instead opting for the middle-ground of "multi-object conditional
> commits".
>
> On Tue, Sep 22, 2026 at 4:10 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi EJ,
> >
> > > 1. Which backend assumptions should we design for? [...]
> >
> > I know I used the term "transaction" a lot during the call, but the real
> > challenge is about establishing an SPI with clear contracts that could be
> > implemented by JDBC and NoSQL persistence using the database features
> > natural to each model.
> >
> > I am fairly confident that a JDBC impl. is possible with Serializable
> > transaction. We should be able to solve all the problems raised in this
> > thread. However, that will make in-memory caching impossible because the
> > database needs to see all the reads (and writes) in a Tx to be able to
> > ensure serializable guarantees. If a cache prevents a read from hitting
> the
> > database, that read (e.g. storage config) will not be a factor in
> > serializability.
> >
> > This connects to my earlier question about the in-memory cache and JDBC
> > (title: Usefulness of InMemoryEntityCache)
> >
> > Are different approaches possible with JDBC? Probably. What is involved?
> > This depends on the SPI contract, I think.
> >
> > > 2. How do we protect the data used by validation? [...]
> >
> > The point mentioned by Robert previously IIRC, is that we need to ground
> > all runtime decisions to a particular state of the catalog.
> >
> > If at commit time the state of the catalog in the database shifts from
> what
> > was used during validation, the request has to restart from scratch.
> >
> > Current JDBC Persistence does this only for relationships between
> entities
> > and grants and between versions of the same entity.
> >
> > Relationships between table locations and storage config are not
> actualized
> > for the purpose of state consistency tracking.
> >
> > From my POV we need to begin by formulating this Persistence state in the
> > Persistence SPI terms. Then, think how we can implement that for JDBC.
> >
> > NoSQL Persistence already has a per-catalog state (not granular), so
> wiring
> > the new SPI to NoSQL should not be too hard, I hope.
> >
> > The state in the SPI does not have to be global to the catalog (as in
> > NoSQL), but it needs to be able to express the idea of which elements
> were
> > engaged during request processing so serve as input to Persistence impl.
> > for cosistency guarantees. If Persistence caches data in memory (for
> > example), it will have to do so in a way that does not violate
> consistency
> > expectations.
> >
> > From my POV we could begin by defining a "Persistence Session" concept.
> > Each request is 1:1 with a session. All reads and writes go through the
> > session. The next step would be the JDBC impl., which probably needs some
> > brainstorming... but I wonder if we could reach a consensus at a high
> level
> > first.
> >
> > WDYT?
> >
> > Thanks,
> > Dmitri.
> >
> > On Tue, Sep 22, 2026 at 5:55 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> > > Thanks for the recap, Dmitri. I'd like to follow up on the three
> points I
> > > raised during the sync. We didn't have much time to discuss them, so it
> > > would help to confirm these before choosing an SPI design.
> > >
> > > 1. Which backend assumptions should we design for?
> > >
> > > Within one Polaris server instance, do we assume entities, grants and
> > > secrets all use the same database backend, or must we support mixing
> > > backends across them?
> > > Must every supported persistence implementation provide transactional
> > > guarantees? Does the database itself need to provide transactions, or
> can
> > > Polaris build that capability on top of simpler database operations?
> > >
> > > I'm happy to work with a homogeneous assumption if that's the intended
> > > scope. Existing NoSQL illustrates the second question: it can publish
> > > changes to multiple objects under one reference using a single
> > conditional
> > > pointer update.
> > >
> > > 2. How do we protect the data used by validation?
> > >
> > > Your storage-configuration and location-overlap examples make the
> problem
> > > concrete. Consider these races:
> > >
> > > A table creation passes validation against the catalog's allowed
> > locations,
> > > then another request changes that configuration before the table is
> > > committed.
> > > Two table-creation requests each check for overlapping locations, both
> > see
> > > no overlap, and then create tables at conflicting locations.
> > >
> > > Checking only the record being written cannot cover these dependencies.
> > The
> > > first also depends on the catalog configuration. The second depends on
> > the
> > > absence of another overlapping table.
> > >
> > > Could the persistence layer protect these reads together with the
> writes,
> > > using backend-internal versions/tokens or database isolation? Shared
> code
> > > would still need to identify the required reads and checks. Could we
> keep
> > > the mechanism for protecting them inside each backend?
> > >
> > > 3. Business rules are duplicated across backend implementations
> > >
> > > Today, the TreeMap
> > > <
> > >
> >
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481
> > > >
> > > and JDBC
> > > <
> > >
> >
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046
> > > >
> > > implementations each load a secret, check its principal, apply the
> > > rotation/reset steps and write it back.
> > >
> > > Changing these business rules therefore requires keeping multiple
> > > implementations in sync. A fix applied to one backend can be missed in
> > > another, causing the same Polaris operation to behave differently
> > depending
> > > on the database.
> > >
> > > My proposal is to implement each business workflow once, against a
> common
> > > set of storage operations with defined consistency and failure
> semantics.
> > > The shared workflow should not need to know whether those operations
> are
> > > implemented through JDBC transactions, NoSQL reference CAS or another
> > > mechanism. Each backend would implement the same storage contract,
> > keeping
> > > its execution details internal. A new database backend could then reuse
> > the
> > > existing business workflows.
> > >
> > > Do these seem like reasonable starting points? We can then evaluate the
> > SPI
> > > against the original failure cases on both JDBC and existing NoSQL.
> > >
> > > -ej
> > >
> > > On Fri, Sep 18, 2026 at 12:10 PM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > wrote:
> > >
> > > > Side note: I just came across [5541], which shows a case where a
> failed
> > > > request leaves persisted side effects.
> > > >
> > > > This can be seen as a coding mistake, of course. However, this is
> about
> > > > executing one request as a single, atomic unit. This is something
> that
> > > > probably needs foundational support and coding patterns in Polaris
> > code.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > > [5541] https://github.com/apache/polaris/pull/5541
> > > >
> > > > On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Here's a recap of the Community Sync discussion as I remember it.
> I'm
> > > > sure
> > > > > my recollection is not complete, so please add your thoughts to
> this
> > > > thread.
> > > > >
> > > > > * Polaris traditionally relies on entity version number to ensure
> > > > > consistency of changes.
> > > > >
> > > > > This works well for internal RBAC grants, which also have version
> > > numbers
> > > > > cross-referenced at the Persistence layer.
> > > > >
> > > > > I do not think this works in more general cases like validating
> > > locations
> > > > > wrt catalog-level storage configuration.
> > > > >
> > > > > Another difficult case involves location overlaps between
> concurrent
> > > > table
> > > > > creations.
> > > > >
> > > > > * We talked about how Persistence calls related to RDBMS
> transactions
> > > (in
> > > > > the JDBC persistence case).
> > > > >
> > > > > There were some concerns raised about whether we need to bind
> > > Persistence
> > > > > API to transactions explicitly.
> > > > >
> > > > > I think it is a valid concern. At the same time, the Persistence
> API
> > > > > should be clear about consistency guarantees across all backend
> > > > > implementations. I think this needs more attention now that the
> > matter
> > > of
> > > > > milti-entioty changes became prominent in [5035].
> > > > >
> > > > > I personally believe that RDBMS-based Persistence in Polaris has to
> > use
> > > > > Serializable transaction isolation. Consequently, the core code
> needs
> > > to
> > > > > provide mechanisms to allow the Persistence implementation to
> > properly
> > > > > connect reads and writes from API requests to JDBC transactions...
> > > > whether
> > > > > we use "transation" as an explicit terms in the SPI or not.
> > > > >
> > > > > Note that currently, the same API request performs reads and writes
> > in
> > > > > _separate_ RDBMS transactions (even separate connections).
> > > > >
> > > > > * We did not discuss Persistence SPI implications in the call, but
> > > > > currently each Persistence call is assumed to be one distinct and
> > > atomic
> > > > > change (please correct me if I'm wrong).
> > > > >
> > > > > Consequently, supporting new use cases involves introducing new SPI
> > > > > methods with many parameters. This can be seen in PR [5035]. This
> > tends
> > > > to
> > > > > overcomplicate the SPI.
> > > > >
> > > > > Ideally, I think the SPI should be simplified to deal with
> > multi-entity
> > > > > and single-entity changes in a coherent manner to avoid any
> ambiguity
> > > > about
> > > > > which method the caller should use.
> > > > >
> > > > > [5035] https://github.com/apache/polaris/pull/5035
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <
> > dimas@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > >> Heads up: This discussion is on the Community Sync call agenda for
> > > Sept
> > > > >> 17.
> > > > >>
> > > > >> Interested parties, please plan to attend, if possible :)
> > > > >>
> > > > >> Cheers,
> > > > >> Dmitri.
> > > > >>
> > > > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi JB,
> > > > >>>
> > > > >>> I've added an agenda item to the next community sync call for
> this
> > > > (Sept
> > > > >>> 17, 2026).
> > > > >>>
> > > > >>> Re: key points: did you mean a real GH discussion or dev email
> > (this
> > > > >>> thread)? Just double checking :) I'm fine with either approach.
> > > > >>>
> > > > >>> Cheers,
> > > > >>> Dmitri.
> > > > >>>
> > > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <
> > > jb@nanthrax.net>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Hi Dmitri,
> > > > >>>>
> > > > >>>> I agree on the need to align on these points, though I'm not
> > > entirely
> > > > >>>> sure a dedicated meeting is necessary. Let's start by using some
> > > time
> > > > >>>> during the next community meeting to discuss it.
> > > > >>>>
> > > > >>>> In the meantime, what if we outline the key points in a GitHub
> > > > >>>> Discussion first? We can then use that as a reference during the
> > > > >>>> meeting.
> > > > >>>>
> > > > >>>> Regards,
> > > > >>>> JB
> > > > >>>>
> > > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <
> > > dimas@apache.org
> > > > >
> > > > >>>> wrote:
> > > > >>>> >
> > > > >>>> > Hi All,
> > > > >>>> >
> > > > >>>> > My impression from this thread is that it might be time for a
> > call
> > > > to
> > > > >>>> > discuss all the related issues and try to arrive at a shared
> > > > >>>> implementation
> > > > >>>> > plan.
> > > > >>>> >
> > > > >>>> > I know meetings are not ideal, but in this case it might be
> > > > >>>> beneficial as a
> > > > >>>> > means for achieving a common understanding of the set of
> > problems
> > > > and
> > > > >>>> > priorities related to this thread. A dedicated metrics meeting
> > > > worked
> > > > >>>> well
> > > > >>>> > from my POV.
> > > > >>>> >
> > > > >>>> > Allocating a time slot in the next community meeting might be
> an
> > > > >>>> option,
> > > > >>>> > although I think we might need the full hour in this case.
> > > > >>>> >
> > > > >>>> > WDYT?
> > > > >>>> >
> > > > >>>> > Thanks,
> > > > >>>> > Dmitri.
> > > > >>>> >
> > > > >>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <
> > > > >>>> prithvisivasankar@gmail.com>
> > > > >>>> > wrote:
> > > > >>>> >
> > > > >>>> > > Hi all,
> > > > >>>> > >
> > > > >>>> > > I rewrote https://github.com/apache/polaris/pull/5035,
> > > following
> > > > >>>> the
> > > > >>>> > > discussion to focus only on the SPI foundation.
> > > > >>>> > >
> > > > >>>> > > What is in the PR now:
> > > > >>>> > >
> > > > >>>> > > - A written manager-level consistency contract in
> > > > >>>> > >
> site/content/in-dev/unreleased/persistence-consistency.md.
> > > > >>>> > > - CAS-aware EntityMutation / GrantMutation records.
> > > > >>>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet,
> > > with
> > > > >>>> backends
> > > > >>>> > > opting in via supportsAtomicMixedCommit().
> > > > >>>> > > - commitChangeSet implementations for the in-memory
> TreeMap
> > > > >>>> backend and
> > > > >>>> > > JDBC.
> > > > >>>> > > - AtomicOperationMetaStoreManager and
> > > > >>>> TransactionalMetaStoreManagerImpl
> > > > >>>> > > grant/revoke paths now build one change set per
> operation,
> > > > >>>> falling back
> > > > >>>> > > to
> > > > >>>> > > individual operations when the backend does not support
> > mixed
> > > > >>>> commits.
> > > > >>>> > > - Tests for the change-set API and TreeMap atomic
> commits.
> > > > >>>> > >
> > > > >>>> > > I intentionally left out:
> > > > >>>> > >
> > > > >>>> > > - Entity deletes in MetaStoreChangeSet.
> > > > >>>> > > - Refactoring createCatalog, dropEntity, and renameEntity
> > to
> > > > use
> > > > >>>> change
> > > > >>>> > > sets.
> > > > >>>> > > - NoSQL support for commitChangeSet.
> > > > >>>> > >
> > > > >>>> > > I am treating this as Phase 1 so the contract and the SPI
> > > > primitive
> > > > >>>> can be
> > > > >>>> > > reviewed before the larger operation migrations land.
> > > > >>>> > >
> > > > >>>> > > Could you please take a look? In particular, I would like to
> > > know
> > > > >>>> whether
> > > > >>>> > > the contract captures the consensus so far, or if it needs
> to
> > > > >>>> address retry
> > > > >>>> > > / external-work coordination before we merge this
> foundation.
> > > > >>>> > >
> > > > >>>> > > Thanks,
> > > > >>>> > > Prithvi S
> > > > >>>> > >
> > > > >>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <
> > > > >>>> dimas@apache.org>
> > > > >>>> > > wrote:
> > > > >>>> > >
> > > > >>>> > > > Hi Robert,
> > > > >>>> > > >
> > > > >>>> > > > I agree that the client-visible operation (e.g. REST API
> > > > request)
> > > > >>>> > > consists
> > > > >>>> > > > of many distinct phases. The database / persistence
> changes
> > > are
> > > > >>>> just one
> > > > >>>> > > of
> > > > >>>> > > > these phases. STS (as an example) is another. My point
> about
> > > > >>>> transactions
> > > > >>>> > > > in JDBC Persistence applies to the former (database
> changes)
> > > > >>>> phase - one
> > > > >>>> > > > retry attempt there should ideally involve exactly one
> > > > >>>> transaction.
> > > > >>>> > > >
> > > > >>>> > > > We certainly need logic inside Polaris Servers to to
> > > coordinate
> > > > >>>> requests
> > > > >>>> > > to
> > > > >>>> > > > various external systems depending on outcomes from
> previous
> > > > >>>> request
> > > > >>>> > > > processing phases.
> > > > >>>> > > >
> > > > >>>> > > > I also agree that validation based on database state
> should
> > be
> > > > >>>> repeated
> > > > >>>> > > > from scratch if we retry the database changes.
> > > > >>>> > > >
> > > > >>>> > > > However, internal RBAC authorization naturally has to
> happen
> > > > >>>> inside the
> > > > >>>> > > > database update phase (read-check-write). IIRC, that is
> part
> > > of
> > > > >>>> the
> > > > >>>> > > > implicit consistency guarantees, which were discussed
> during
> > > > early
> > > > >>>> > > project
> > > > >>>> > > > months (but never got written down in full clarity,
> > > > >>>> unfortunately). If
> > > > >>>> > > the
> > > > >>>> > > > Authorization check is inside that phase for internal
> RBAC,
> > I
> > > > >>>> suppose it
> > > > >>>> > > > will be there for other Authorizers too and that can
> involve
> > > > >>>> external
> > > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly expect
> > > > >>>> authorization to be
> > > > >>>> > > > efficient, but I'm not sure how quick it can be in
> practice.
> > > > >>>> There's
> > > > >>>> > > > certainly potential for delays in some (perhaps
> infrequent)
> > > > cases.
> > > > >>>> > > >
> > > > >>>> > > > Cheers,
> > > > >>>> > > > Dmitri.
> > > > >>>> > > >
> > > > >>>> > > >
> > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <
> > snazy@snazy.de>
> > > > >>>> wrote:
> > > > >>>> > > >
> > > > >>>> > > > > Hi Dmitri,
> > > > >>>> > > > >
> > > > >>>> > > > > I agree that one logical operation needs a consistent
> view
> > > of
> > > > >>>> the whole
> > > > >>>> > > > > backend state.
> > > > >>>> > > > > What worries me is treating one long JDBC transaction as
> > the
> > > > >>>> boundary
> > > > >>>> > > of
> > > > >>>> > > > > that operation and then retrying the complete request if
> > the
> > > > >>>> commit
> > > > >>>> > > > fails.
> > > > >>>> > > > >
> > > > >>>> > > > > The request can also write metadata to object storage or
> > > call
> > > > >>>> STS.
> > > > >>>> > > > > Each system can leave us with an outcome whose certainty
> > we
> > > > >>>> cannot
> > > > >>>> > > know.
> > > > >>>> > > > > The database may have committed before the connection
> was
> > > > lost,
> > > > >>>> an
> > > > >>>> > > > > object-store write may have succeeded before a timeout,
> or
> > > STS
> > > > >>>> may have
> > > > >>>> > > > > issued credentials before its response was lost.
> > > > >>>> > > > > A database rollback cannot undo any of those effects,
> and
> > > > >>>> retrying the
> > > > >>>> > > > > whole request may repeat them.
> > > > >>>> > > > >
> > > > >>>> > > > > So I think the contract has to separate the overall
> > > operation
> > > > >>>> from each
> > > > >>>> > > > > attempt to commit it.
> > > > >>>> > > > > Each database or backend commit attempt should stay
> short.
> > > > >>>> > > > > When we retry, validation and authorization need to use
> > the
> > > > >>>> state for
> > > > >>>> > > > that
> > > > >>>> > > > > new attempt, and we need clear rules for which external
> > work
> > > > >>>> can be
> > > > >>>> > > > reused,
> > > > >>>> > > > > repeated, cleaned up, or reconciled after an uncertain
> > > > outcome.
> > > > >>>> > > > >
> > > > >>>> > > > > Some contextual state could be useful for carrying that
> > > state.
> > > > >>>> > > > > But the JDBC connection and transaction should be an
> > > > >>>> implementation
> > > > >>>> > > > detail
> > > > >>>> > > > > inside it, not the boundary of the complete operation.
> > > > >>>> > > > >
> > > > >>>> > > > > Cheers,
> > > > >>>> > > > > Robert
> > > > >>>> > > > >
> > > > >>>> > > > >
> > > > >>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <
> > > > >>>> dimas@apache.org
> > > > >>>> > > >
> > > > >>>> > > > > wrote:
> > > > >>>> > > > >
> > > > >>>> > > > > > Hi All,
> > > > >>>> > > > > >
> > > > >>>> > > > > > The idea of a backend-agnostic change-set primitive
> > sounds
> > > > >>>> good.
> > > > >>>> > > > > >
> > > > >>>> > > > > > However, I am not sure it is sufficient for all the
> use
> > > > cases
> > > > >>>> we
> > > > >>>> > > > touched
> > > > >>>> > > > > > here. More specifically, the state read by validation
> > code
> > > > is
> > > > >>>> not
> > > > >>>> > > > > > necessarily reflected in the change set:
> > > > >>>> > > > > >
> > > > >>>> > > > > > * Unchanged entities might be considered by
> validation,
> > > but
> > > > >>>> changed
> > > > >>>> > > in
> > > > >>>> > > > a
> > > > >>>> > > > > > parallel request.
> > > > >>>> > > > > > * Even for changed entities the state read by
> validation
> > > may
> > > > >>>> differ
> > > > >>>> > > > from
> > > > >>>> > > > > > the state being altered. Some validation code talks to
> > the
> > > > >>>> MetaStore
> > > > >>>> > > > > > directly, outside of the data produced by the
> Resolver.
> > > > >>>> > > > > >
> > > > >>>> > > > > > I do not think Polaris offers any explicit mechanisms
> > > (ATM)
> > > > >>>> to ensure
> > > > >>>> > > > > > consistency between these reads and subsequent writes.
> > > > >>>> > > > > >
> > > > >>>> > > > > > As far as JDBC goes, running an overarching
> Transaction
> > > > >>>> across all
> > > > >>>> > > > reads
> > > > >>>> > > > > > and writes in the same request, at the SERIALIZABLE
> > > > isolation
> > > > >>>> level
> > > > >>>> > > in
> > > > >>>> > > > > the
> > > > >>>> > > > > > backing RDBMS could solve the problem, I think.
> > > > >>>> > > > > >
> > > > >>>> > > > > > Indeed, such a transaction might be long to
> accommodate
> > > > calls
> > > > >>>> made by
> > > > >>>> > > > > > Polaris to external storage, etc. However, is that a
> > > > problem?
> > > > >>>> I think
> > > > >>>> > > > the
> > > > >>>> > > > > > alternative is for the JDBC Persistence impl. to
> > > "manually"
> > > > >>>> track all
> > > > >>>> > > > > reads
> > > > >>>> > > > > > under the same request and redo them in the small
> > > > transaction
> > > > >>>> that
> > > > >>>> > > > > persists
> > > > >>>> > > > > > the writes. This will also add RDBMS overhead and
> > require
> > > > >>>> complex
> > > > >>>> > > code
> > > > >>>> > > > in
> > > > >>>> > > > > > Polaris to handle the data properly.
> > > > >>>> > > > > >
> > > > >>>> > > > > > Tracking the request-wide transaction can be done only
> > for
> > > > >>>> JDBC
> > > > >>>> > > without
> > > > >>>> > > > > > leaking "transaction" concepts to the NoSQL
> > Persistence, I
> > > > >>>> think.
> > > > >>>> > > NoSQL
> > > > >>>> > > > > > will use other mechanisms to ensure read/write
> > > consistency.
> > > > >>>> > > > > >
> > > > >>>> > > > > > Connecting to Robert's email (a parallel branch in
> this
> > > > >>>> discussion),
> > > > >>>> > > > I'd
> > > > >>>> > > > > > like to propose this approach:
> > > > >>>> > > > > >
> > > > >>>> > > > > > * Each request establishes a "Data Context"
> > > > >>>> > > > > > - In JDBC the Data Context corresponds to a JDBC
> > > > Connection
> > > > >>>> + Tx
> > > > >>>> > > > > > - In NoSQL the Data Context tracks one or more
> > reference
> > > > >>>> hashes
> > > > >>>> > > > > > * All Persistence access in the same request goes
> > through
> > > > the
> > > > >>>> same
> > > > >>>> > > Data
> > > > >>>> > > > > > Context
> > > > >>>> > > > > > * All Persistence changes are committed once at the
> end
> > of
> > > > the
> > > > >>>> > > request
> > > > >>>> > > > > > - Not all changes have to be globally atomic. For
> > > example,
> > > > >>>> changes
> > > > >>>> > > in
> > > > >>>> > > > > > Catalogs A and B do not have to be atomic with respect
> > to
> > > > >>>> each other.
> > > > >>>> > > > We
> > > > >>>> > > > > > can go deeper into this later. This is relevant to
> > NoSQL.
> > > > >>>> > > > > > - Transactional backends like JDBC can, of course,
> > > choose
> > > > >>>> to make
> > > > >>>> > > all
> > > > >>>> > > > > > changes globally atomic.
> > > > >>>> > > > > > * A commit can fail in two main ways:
> > > > >>>> > > > > > - A retriable failure like an optimistic lock error
> or
> > > Tx
> > > > >>>> > > > > serializability
> > > > >>>> > > > > > error
> > > > >>>> > > > > > - A non-triable logical error (e.g. entity not
> found)
> > > > >>>> > > > > > * On a retriable error the whole request is
> re-attempted
> > > (as
> > > > >>>> if
> > > > >>>> > > > > resubmitted
> > > > >>>> > > > > > by a client) a few times (configurable timeout).
> > > > >>>> > > > > >
> > > > >>>> > > > > > WDYT?
> > > > >>>> > > > > >
> > > > >>>> > > > > > Cheers,
> > > > >>>> > > > > > Dmitri.
> > > > >>>> > > > > >
> > > > >>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré
> <
> > > > >>>> > > jb@nanthrax.net
> > > > >>>> > > > >
> > > > >>>> > > > > > wrote:
> > > > >>>> > > > > >
> > > > >>>> > > > > > > Hi all
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > I think Robert has a good point.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > If the atomicity guarantee lives only in
> > > BasePersistence,
> > > > >>>> then the
> > > > >>>> > > > > > > manager contract can't tell a caller whether the
> state
> > > it
> > > > >>>> read for
> > > > >>>> > > > > > > validation/authorization/credential-vending is the
> > same
> > > > >>>> state that
> > > > >>>> > > > > > > eventually commits. That's the actual bug class
> behind
> > > the
> > > > >>>> JDBC
> > > > >>>> > > > > > > symptoms (not any single operation being non-atomic,
> > but
> > > > the
> > > > >>>> > > contract
> > > > >>>> > > > > > > being silent about it. Every operation-specific fix
> > > (like
> > > > >>>> #4939,
> > > > >>>> > > > #5035
> > > > >>>> > > > > > > or #5095) re-answers this question locally and it
> > keeps
> > > > >>>> recurring.
> > > > >>>> > > So
> > > > >>>> > > > > > > the deliverable should start with a written
> > consistency
> > > > >>>> contract at
> > > > >>>> > > > > > > the manager level.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > I'm not sure migrating everyone to
> > > > >>>> > > TransactionalMetaStoreManagerImpl
> > > > >>>> > > > > > > is a good idea. It would tie the logical change set
> > to a
> > > > DB
> > > > >>>> > > > > > > transaction spanning the REST request. It means:
> > > > >>>> > > > > > > - it holds a durable transaction open across slow
> > > external
> > > > >>>> work
> > > > >>>> > > > > > > (credential vending, OPA, Ranger, ...)
> > > > >>>> > > > > > > - it doesn't map to NoSQL
> > > > >>>> > > > > > > - It wraps single-row updates in
> > runWiithinTransaction,
> > > > >>>> which is a
> > > > >>>> > > > > > overhead
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > So, I think the transactional manager isn't a
> portable
> > > > >>>> target. It's
> > > > >>>> > > > > > > "only" one backend's strategy.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > I think Privthi's approach is right. A
> > backend-agnostic
> > > > >>>> change-set
> > > > >>>> > > > > > > primitive with a documented fallback is the correct
> > > shape.
> > > > >>>> There is
> > > > >>>> > > > > > > one caveat: without carrying the original entitiy
> (not
> > > > just
> > > > >>>> the new
> > > > >>>> > > > > > > one) the change set can't express optimistic
> > > concurrency.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > I propose the following multi-steps approach:
> > > > >>>> > > > > > > 1. We write the manager-level consistency contract
> as
> > > > >>>> Robert asked.
> > > > >>>> > > > It
> > > > >>>> > > > > > > should include the explicit statement that a logical
> > > > change
> > > > >>>> set is
> > > > >>>> > > > not
> > > > >>>> > > > > > > a request-scope DB transaction.
> > > > >>>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency
> > > > pattern)
> > > > >>>> > > baseline
> > > > >>>> > > > > > > first-class in EntityMutation before merging the SPI
> > > > >>>> > > > > > > 3. Then we refactor
> > > createCatalog/dropEntity/renameEntity.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > Thoughts?
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > Regards
> > > > >>>> > > > > > > JB
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <
> > > > >>>> snazy@snazy.de>
> > > > >>>> > > wrote:
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > Hi all,
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > Yufei's clarification about where the atomicity
> > > > guarantee
> > > > >>>> is
> > > > >>>> > > > defined
> > > > >>>> > > > > > > seems
> > > > >>>> > > > > > > > important.
> > > > >>>> > > > > > > > If it is a property of the lower-level
> > BasePersistence
> > > > >>>> contract
> > > > >>>> > > > > rather
> > > > >>>> > > > > > > than
> > > > >>>> > > > > > > > the general PolarisMetaStoreManager contract, the
> > > > general
> > > > >>>> > > contract
> > > > >>>> > > > > does
> > > > >>>> > > > > > > not
> > > > >>>> > > > > > > > tell callers whether the state used for
> validation,
> > > > >>>> > > authorization,
> > > > >>>> > > > or
> > > > >>>> > > > > > > > credential vending is consistent with the change
> > that
> > > > >>>> eventually
> > > > >>>> > > > > > commits.
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > The current PRs suggest that operation-specific
> > > > >>>> multi-object
> > > > >>>> > > > methods
> > > > >>>> > > > > > can
> > > > >>>> > > > > > > > fix individual cases, while leaving the same
> > contract
> > > > >>>> question to
> > > > >>>> > > > > recur
> > > > >>>> > > > > > > for
> > > > >>>> > > > > > > > each new case.
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > I would also avoid defining a logical change set
> as
> > a
> > > > >>>> database
> > > > >>>> > > > > > > transaction
> > > > >>>> > > > > > > > around an entire REST request.
> > > > >>>> > > > > > > > That would tie the contract to the backend and
> > > database,
> > > > >>>> and
> > > > >>>> > > could
> > > > >>>> > > > > keep
> > > > >>>> > > > > > > the
> > > > >>>> > > > > > > > durable attempt open across slow external work.
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > So is the choice really between the two current
> > > manager
> > > > >>>> > > > > > implementations,
> > > > >>>> > > > > > > or
> > > > >>>> > > > > > > > do we first need to revisit the boundary and
> > > guarantees
> > > > >>>> exposed
> > > > >>>> > > to
> > > > >>>> > > > > > their
> > > > >>>> > > > > > > > callers?
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > Cheers,
> > > > >>>> > > > > > > > Robert
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri
> > Bourlatchkov <
> > > > >>>> > > > > dimas@apache.org
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > > wrote:
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > > Hi Yufei,
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Thanks for the link. I stand corrected. The "one
> > > > atomic
> > > > >>>> change
> > > > >>>> > > > per
> > > > >>>> > > > > > > method"
> > > > >>>> > > > > > > > > contract as defined in javadoc does apply only
> to
> > > > >>>> > > BasePersistence
> > > > >>>> > > > > > > > > and AtomicOperationMetaStoreManager (which
> > delegates
> > > > to
> > > > >>>> > > > > > > BasePersistence).
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Note that all non-test Persistence
> implementations
> > > in
> > > > >>>> the
> > > > >>>> > > Polaris
> > > > >>>> > > > > > > codebase
> > > > >>>> > > > > > > > > extend those classes (which is probably why I
> was
> > > > >>>> confused
> > > > >>>> > > about
> > > > >>>> > > > > > > atomicity
> > > > >>>> > > > > > > > > expectations).
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > However, this creates a gap in the Persistence
> SPI
> > > > >>>> > > specification.
> > > > >>>> > > > > If
> > > > >>>> > > > > > > > > other PolarisMetaStoreManager implementations do
> > not
> > > > >>>> have to
> > > > >>>> > > > comply
> > > > >>>> > > > > > > with
> > > > >>>> > > > > > > > > this principle, it will create a conceptual
> > > difficulty
> > > > >>>> at call
> > > > >>>> > > > > sites.
> > > > >>>> > > > > > > How
> > > > >>>> > > > > > > > > can PolarisMetaStoreManager callers reason about
> > > > >>>> consistency
> > > > >>>> > > and
> > > > >>>> > > > > > > durability
> > > > >>>> > > > > > > > > behaviours in general?
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > I believe we need to address that as part of
> this
> > > > >>>> discussion.
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Cheers,
> > > > >>>> > > > > > > > > Dmitri.
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <
> > > > >>>> flyrain000@gmail.com
> > > > >>>> > > >
> > > > >>>> > > > > > wrote:
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with
> > the
> > > > >>>> idea that
> > > > >>>> > > one
> > > > >>>> > > > > > > method
> > > > >>>> > > > > > > > > call
> > > > >>>> > > > > > > > > > means one atomic change.
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > If the "MetaStore SPI" refers to the interface
> > > > >>>> > > > > > > PolarisMetaStoreManager, I
> > > > >>>> > > > > > > > > > don't think we've ever state each method to be
> > > > >>>> atomic. We did
> > > > >>>> > > > > > clarify
> > > > >>>> > > > > > > > > > atomicity[1] in the interface BasePersistence
> > > > though.
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > 1.
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>>
> > > >
> > >
> >
> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > Yufei
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri
> > > > Bourlatchkov <
> > > > >>>> > > > > > > dimas@apache.org>
> > > > >>>> > > > > > > > > > wrote:
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > > Hi Yufei,
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > I agree that JDBC transactions must be
> handled
> > > > more
> > > > >>>> > > > explicitly.
> > > > >>>> > > > > > > > > However,
> > > > >>>> > > > > > > > > > > I'm not sure that simply moving to
> > > > >>>> > > > > > > TransactionalMetaStoreManagerImpl is
> > > > >>>> > > > > > > > > > > sufficient.
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with
> > the
> > > > >>>> idea that
> > > > >>>> > > one
> > > > >>>> > > > > > > method
> > > > >>>> > > > > > > > > call
> > > > >>>> > > > > > > > > > > means one atomic change [1]. The
> > "transactional"
> > > > >>>> MetaStore
> > > > >>>> > > > > impl.
> > > > >>>> > > > > > is
> > > > >>>> > > > > > > > > but a
> > > > >>>> > > > > > > > > > > sub-case of that. It cannot alter the
> > high-level
> > > > >>>> contract.
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > We could add SPI methods having multiple
> > object
> > > > >>>> parameters
> > > > >>>> > > to
> > > > >>>> > > > > > > represent
> > > > >>>> > > > > > > > > > > grouped changes, but I am not sure it will
> be
> > a
> > > > >>>> sound
> > > > >>>> > > design.
> > > > >>>> > > > > > This
> > > > >>>> > > > > > > will
> > > > >>>> > > > > > > > > > > bloat the interface surfaces and require
> extra
> > > > >>>> impl. effort
> > > > >>>> > > > for
> > > > >>>> > > > > > > each
> > > > >>>> > > > > > > > > > > backend type. More importantly, adding
> > multi-arg
> > > > >>>> change
> > > > >>>> > > > methods
> > > > >>>> > > > > > > still
> > > > >>>> > > > > > > > > > won't
> > > > >>>> > > > > > > > > > > address the problem of reads being
> consistent
> > > with
> > > > >>>> writes,
> > > > >>>> > > > > > because
> > > > >>>> > > > > > > each
> > > > >>>> > > > > > > > > > > method call will still be independent
> > regarding
> > > > the
> > > > >>>> data
> > > > >>>> > > > stored
> > > > >>>> > > > > > in
> > > > >>>> > > > > > > the
> > > > >>>> > > > > > > > > > > database.
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > I tend to think we need to introduce a
> "change
> > > > set"
> > > > >>>> or
> > > > >>>> > > > "atomic
> > > > >>>> > > > > > > batch"
> > > > >>>> > > > > > > > > > > concept to core Persistence and associate
> each
> > > > REST
> > > > >>>> API
> > > > >>>> > > > request
> > > > >>>> > > > > > > with
> > > > >>>> > > > > > > > > one
> > > > >>>> > > > > > > > > > > such change set, which will be committed (or
> > > > rolled
> > > > >>>> back)
> > > > >>>> > > at
> > > > >>>> > > > > the
> > > > >>>> > > > > > > end of
> > > > >>>> > > > > > > > > > the
> > > > >>>> > > > > > > > > > > request. I believe Ayush mentioned a similar
> > > > >>>> concept in PR
> > > > >>>> > > > 4939
> > > > >>>> > > > > > > [2]. In
> > > > >>>> > > > > > > > > > > JDBC each change set will naturally be
> > > associated
> > > > >>>> with an
> > > > >>>> > > > RDBMS
> > > > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each
> atomic
> > > > >>>> change set
> > > > >>>> > > > will
> > > > >>>> > > > > be
> > > > >>>> > > > > > > > > > > associated with one CAS operation on the
> > > > underlying
> > > > >>>> > > database.
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > [1]
> > > > >>>> > > > > > >
> > > > >>>>
> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > [2]
> > > > >>>> > > > > > >
> > > > >>>>
> > https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > Cheers,
> > > > >>>> > > > > > > > > > > Dmitri.
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
> > > > >>>> > > > flyrain000@gmail.com
> > > > >>>> > > > > >
> > > > >>>> > > > > > > wrote:
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > Thanks for raising this, Dmitri.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > These are valid concerns, and they were
> > > already
> > > > >>>> > > recognized
> > > > >>>> > > > > when
> > > > >>>> > > > > > > we
> > > > >>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At
> > > that
> > > > >>>> time, we
> > > > >>>> > > > > chose
> > > > >>>> > > > > > > to use
> > > > >>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the
> JDBC
> > > due
> > > > >>>> to the
> > > > >>>> > > > > > > simplicity. I
> > > > >>>> > > > > > > > > > > think
> > > > >>>> > > > > > > > > > > > most of the issues mentioned here can
> > already
> > > be
> > > > >>>> > > addressed
> > > > >>>> > > > by
> > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > For example, rename is already wrapped in
> a
> > > > >>>> transaction
> > > > >>>> > > in
> > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1].
> > > > Similarly,
> > > > >>>> catalog
> > > > >>>> > > > > > > creation,
> > > > >>>> > > > > > > > > > which
> > > > >>>> > > > > > > > > > > > involves reading and creating multiple
> > > objects,
> > > > >>>> is also
> > > > >>>> > > > > > executed
> > > > >>>> > > > > > > > > > within a
> > > > >>>> > > > > > > > > > > > transaction [2].
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > I see two possible directions:
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > 1.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager
> > > > >>>> together with
> > > > >>>> > > the
> > > > >>>> > > > > > > > > persistence
> > > > >>>> > > > > > > > > > > > backends (such as JDBC) to provide the
> > > > required
> > > > >>>> > > > > consistency
> > > > >>>> > > > > > > > > > guarantees
> > > > >>>> > > > > > > > > > > > for
> > > > >>>> > > > > > > > > > > > specific operations, similar to what
> > > > >>>> > > > > > > > > > TransactionalMetaStoreManagerImpl
> > > > >>>> > > > > > > > > > > > does.
> > > > >>>> > > > > > > > > > > > 2.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > Migrate the persistence backends (such
> as
> > > > >>>> JDBC) to use
> > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl
> > directly.
> > > > We
> > > > >>>> may
> > > > >>>> > > have
> > > > >>>> > > > to
> > > > >>>> > > > > > > deal
> > > > >>>> > > > > > > > > with
> > > > >>>> > > > > > > > > > > > transactional semantic mismatches
> across
> > > > >>>> different
> > > > >>>> > > > > > persistence
> > > > >>>> > > > > > > > > > > backends.
> > > > >>>> > > > > > > > > > > > For example, we would likely avoid
> using
> > > > JDBC's
> > > > >>>> > > > > > > > > > `runWithinTransaction`
> > > > >>>> > > > > > > > > > > > for
> > > > >>>> > > > > > > > > > > > single row updates, which adds
> additional
> > > > >>>> overhead and
> > > > >>>> > > > > > > complexity
> > > > >>>> > > > > > > > > > > > without
> > > > >>>> > > > > > > > > > > > benefits.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > References:
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > 1.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>>
> > > >
> > >
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > > > >>>> > > > > > > > > > > > 2.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>>
> > > >
> > >
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > Yufei
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri
> > > > >>>> Bourlatchkov <
> > > > >>>> > > > > > > > > dimas@apache.org>
> > > > >>>> > > > > > > > > > > > wrote:
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > Hi all,
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a
> > > > couple
> > > > >>>> of
> > > > >>>> > > > > > interesting
> > > > >>>> > > > > > > PRs:
> > > > >>>> > > > > > > > > > > > > [4939], [5035].
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > It looks like people are starting to
> > > encounter
> > > > >>>> > > > consistency
> > > > >>>> > > > > > > issues
> > > > >>>> > > > > > > > > in
> > > > >>>> > > > > > > > > > > > > JDBC persistence.
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight into
> the
> > > > >>>> underlying
> > > > >>>> > > > > issues.
> > > > >>>> > > > > > > They
> > > > >>>> > > > > > > > > > offer
> > > > >>>> > > > > > > > > > > > > incremental fixes that can work.
> However,
> > I
> > > > >>>> believe it
> > > > >>>> > > is
> > > > >>>> > > > > > time
> > > > >>>> > > > > > > for
> > > > >>>> > > > > > > > > > the
> > > > >>>> > > > > > > > > > > > > Polaris community to review and improve
> > this
> > > > >>>> area of
> > > > >>>> > > the
> > > > >>>> > > > > > > codebase
> > > > >>>> > > > > > > > > > > > > holistically.
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > By this, I mean finding a solution that
> > can
> > > be
> > > > >>>> applied
> > > > >>>> > > to
> > > > >>>> > > > > all
> > > > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC,
> > > NoSQL)
> > > > >>>> and
> > > > >>>> > > > addresses
> > > > >>>> > > > > > > these
> > > > >>>> > > > > > > > > > > > > aspects:
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent
> > > changes
> > > > >>>> where
> > > > >>>> > > the
> > > > >>>> > > > > > > service
> > > > >>>> > > > > > > > > > reads
> > > > >>>> > > > > > > > > > > > > and validates current catalog state,
> > then
> > > > >>>> commits a
> > > > >>>> > > > > change
> > > > >>>> > > > > > > (e.g.
> > > > >>>> > > > > > > > > > > > > name clashes during renames).
> > > > >>>> > > > > > > > > > > > > * Supporting consistent but independent
> > > > changes
> > > > >>>> to RBAC
> > > > >>>> > > > > > grants
> > > > >>>> > > > > > > and
> > > > >>>> > > > > > > > > > > > > MetaStore entities. This independence
> is
> > > > >>>> needed to
> > > > >>>> > > > > support
> > > > >>>> > > > > > > > > > > > > external authorizers like OPA and
> > Ranger.
> > > > >>>> > > > > > > > > > > > > * Supporting atomic changes across
> > multiple
> > > > >>>> similar
> > > > >>>> > > > > entities.
> > > > >>>> > > > > > > > > > > > > * Supporting authorization-based
> filtering
> > > of
> > > > >>>> list
> > > > >>>> > > > > operations
> > > > >>>> > > > > > > (cf.
> > > > >>>> > > > > > > > > > > > > [4831]).
> > > > >>>> > > > > > > > > > > > > * Supporting credential-vending
> decisions
> > > that
> > > > >>>> are
> > > > >>>> > > rooted
> > > > >>>> > > > > in
> > > > >>>> > > > > > > the
> > > > >>>> > > > > > > > > > > > > exact state of the catalog.
> > > > >>>> > > > > > > > > > > > > * Supporting server-side retries for
> > > transient
> > > > >>>> > > > persistence
> > > > >>>> > > > > > > failures
> > > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability
> > failures).
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > Please share your comments and ideas.
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > [4831]
> > > > >>>> https://github.com/apache/polaris/pull/4831
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > [4939]
> > > > >>>> https://github.com/apache/polaris/pull/4939
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > [5035]
> > > > >>>> https://github.com/apache/polaris/pull/5035
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > Thanks,
> > > > >>>> > > > > > > > > > > > > Dmitri
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>>
> > > > >>>
> > > >
> > >
> >
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi All,
It's good to see this discussion progressing :)
I do not insist on strongly binding REST requests to JDBC (or other)
transactions.
However, I think we have to keep in mind that the Persistence SPI should be
implementable via JDBC.
With that in mind, a trivial CAS on multiple changed rows may not be
sufficient. For example, if we change table A, but have to read entities
for tables B and C to check for location overlaps and B and C are served
from memory (cache) in node X but are updated in parallel by node Y, how
will Persistence be able to detect that conflict? Even "UPDATE where
version = ?" for each of the changed entities does not guarantee that the
database will detect the conflict. Not even with serializable Tx isolation.
Serializable isolation might help, but then the cache always has to send
all reads through to the database. That is, on every retry.
Right now this is achieved by a combination of the Entity Cache and
Resolver logic. However, the need to read through stems only from knowing a
specific database implementation (RDBMS with Serializable isolation). So,
the Cache and Resolver apparently make certain assumptions about that.
At the same time there are call paths into Persistence that do not involve
the Resolver. Those paths are not covered by the (implicit) read-through
guarantees of the Resolver (and Entity Cache) code.
It would be nice to abstract those aspects so that higher level (REST
service) code could stay agnostic of Persistence technology and Persistence
_implementation_ would then deal with caching and all database-specific
aspects.
More concretely, I think read-through for every entity does not have to be
forced by the high-level code (where the Resolver currently lives). It is a
concern of the database-specific layer.
WDYT?
Thanks,
Dmitri.
On Thu, Oct 1, 2026 at 12:14 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Dennis
>
> I agree with you. I think "multi object conditional commits" rather
> than a request-scoped transaction is aligned with what Robert and I
> argued: commit attempts stay short, it maps to both JDBC and NoSQL. An
> explicit read set checked at commit time stays compatible with
> in-memory caching (I don't think a serializable only approach does
> not).
>
> I believe it is also a small step forward from what we already have.
> The multi-table commit path already does a multi-entity CAS through
> updateEntitiesPropertiesIfNotChanged. We should extend it.
>
> I have some concerns about using PolarisResolutionManifest as is:
> 1. As Dmitri mentioned, the location overlap check does not go through
> the manifest at all. So the commit "contract" needs predicate checks
> that the backend should re-evaluate at commit time.
> 2. The manifest is not an "open book" today.
> getPassthroughResolvedPath creates a single user Resolver on every
> call and does not record the version it returned. I don't see how to
> store the version easily today.
> 3. In your example the catalog is not pinned, but validating a table
> location against the allowed locations depends on the catalog storage
> configuration, so for a create it would have to be. I would prefer
> recording versions on read, or pinning by default with an explicit
> opt-out.
> 4. To simplify, I would define the read set as a small type in the
> Persistence SPI, which the manifest produces, rather than making the
> manifest itself the SPI.
>
> I would propose to define the read set in Persistence SPI terms,
> independent from the resolver. and also include predicate (location
> overlap, name conflict) in the contract.
>
> I think it's pretty close to what Dmitri is proposing (the
> "Persistence Session "iidea).
>
> Regards
> JB
>
> On Thu, Oct 1, 2026 at 2:41 AM Dennis Huo <dh...@apache.org> wrote:
> >
> > Agree we should formulate in terms of Persistence SPIs first.
> >
> > IIRC we talked about this in one of the community syncs but I forgot to
> > bring it here -- the PolarisResolutionManifest in concept already
> > represents *exactly* the set of "which elements were engaged during
> request
> > processing".
> >
> > Adherence may have drifted, but at least early on, we made sure all the
> > IcebergCatalog logic did *not* get to do new direct reads from the
> > persistence layer, but instead needs to "read" from the *authorized* set
> of
> > entities contained in a PolarisResolutionManifest (or declared as a
> > "passthroughPath" == allow reading fresh from DB but had to pre-register
> it
> > for authz purposes).
> >
> > It's important to keep this unified anyways because it really serves two
> > things:
> >
> > 1. Declares the set of entities whose state you expect to
> resolve/snapshot
> > up-front in the operation-handling, that the IcebergCatalog logic then
> > operates on later; also explicitly declare which ones require fetching
> > *fresh* versions mid-operation (addPassthroughPath)
> > 2. Enforces that RBAC authz is performed against that pre-declared set of
> > entities -- if later processign were allowed to access arbitrary other
> > entities to make handler decisions, then how did we "prove" that the
> > authorization engine actually allowed the request to access those
> arbitrary
> > other entities? Even if it's just for internal-processing purposes, the
> > very fact that something got accessed means there's a route to accidental
> > leakage of unauthorized data.
> >
> > So in a way, solid/provable authz and solid/provable consistency go
> > hand-in-hand.
> >
> > I think PolarisResolutionManifest in its current form is probably a bit
> > opaque and doesn't 100% convey the semantics we want quite yet, but it
> > seems like the right SPI starting point to extend or at least evolve
> from.
> > Basically, aside from addPath and addPassthroughPath, there could be a
> > notion of annotating a given path as being one of the "dependent states"
> to
> > carry over into the later commit. Conceptually:
> >
> > // Prepare operation processing
> > resolutionManifest.prefetch(rootCatalog, mustBeUnchangedAtCommitTime =
> > false); // For this operation let's pretend we don't care if the root
> > catalog changes before we commit the change to the tables
> > resolutionManifest.prefetch(parentNamespace, mustBeUnchangedAtCommitTime
> =
> > false); // For this operation let's pretend we don't care if the parent
> > namespace changes before we commit the change to the tables
> > resolutionManifest.prefetch(table1, mustBeUnchangedAtCommitTime = true);
> > // Assume multi-table commit - table1 will have changes and also must
> not
> > have already changed at commit time
> > resolutionManifest.prefetch(table2, mustBeUnchangedAtCommitTime = true);
> > // Same for table2
> >
> > // .... Do all the Iceberg processing work
> >
> > // When we commit, we use the resolutionManifest as the ledger of things
> we
> > used that must still be at their unchanged version at time of commit
> > metastore.doConditionalBatchOmmit(List.of(table1Updated, table2Updated),
> > resolutionManifest.getSetOfUnchangedEntityRequirements());
> >
> > Overall this fits the optimistic-concurrency theme of Iceberg and Polaris
> > in general, and would *intentionally* not try to express a full-fledged
> > "Transaction" session in which you do a bunch of
> reads/writes/reads/writes
> > on a single entity as if you're operating in an uncommitted block on a
> > RDBMS, instead opting for the middle-ground of "multi-object conditional
> > commits".
> >
> > On Tue, Sep 22, 2026 at 4:10 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi EJ,
> > >
> > > > 1. Which backend assumptions should we design for? [...]
> > >
> > > I know I used the term "transaction" a lot during the call, but the
> real
> > > challenge is about establishing an SPI with clear contracts that could
> be
> > > implemented by JDBC and NoSQL persistence using the database features
> > > natural to each model.
> > >
> > > I am fairly confident that a JDBC impl. is possible with Serializable
> > > transaction. We should be able to solve all the problems raised in this
> > > thread. However, that will make in-memory caching impossible because
> the
> > > database needs to see all the reads (and writes) in a Tx to be able to
> > > ensure serializable guarantees. If a cache prevents a read from
> hitting the
> > > database, that read (e.g. storage config) will not be a factor in
> > > serializability.
> > >
> > > This connects to my earlier question about the in-memory cache and JDBC
> > > (title: Usefulness of InMemoryEntityCache)
> > >
> > > Are different approaches possible with JDBC? Probably. What is
> involved?
> > > This depends on the SPI contract, I think.
> > >
> > > > 2. How do we protect the data used by validation? [...]
> > >
> > > The point mentioned by Robert previously IIRC, is that we need to
> ground
> > > all runtime decisions to a particular state of the catalog.
> > >
> > > If at commit time the state of the catalog in the database shifts from
> what
> > > was used during validation, the request has to restart from scratch.
> > >
> > > Current JDBC Persistence does this only for relationships between
> entities
> > > and grants and between versions of the same entity.
> > >
> > > Relationships between table locations and storage config are not
> actualized
> > > for the purpose of state consistency tracking.
> > >
> > > From my POV we need to begin by formulating this Persistence state in
> the
> > > Persistence SPI terms. Then, think how we can implement that for JDBC.
> > >
> > > NoSQL Persistence already has a per-catalog state (not granular), so
> wiring
> > > the new SPI to NoSQL should not be too hard, I hope.
> > >
> > > The state in the SPI does not have to be global to the catalog (as in
> > > NoSQL), but it needs to be able to express the idea of which elements
> were
> > > engaged during request processing so serve as input to Persistence
> impl.
> > > for cosistency guarantees. If Persistence caches data in memory (for
> > > example), it will have to do so in a way that does not violate
> consistency
> > > expectations.
> > >
> > > From my POV we could begin by defining a "Persistence Session" concept.
> > > Each request is 1:1 with a session. All reads and writes go through the
> > > session. The next step would be the JDBC impl., which probably needs
> some
> > > brainstorming... but I wonder if we could reach a consensus at a high
> level
> > > first.
> > >
> > > WDYT?
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Tue, Sep 22, 2026 at 5:55 PM EJ Wang <
> ej.wang.dev.interest@gmail.com>
> > > wrote:
> > >
> > > > Thanks for the recap, Dmitri. I'd like to follow up on the three
> points I
> > > > raised during the sync. We didn't have much time to discuss them, so
> it
> > > > would help to confirm these before choosing an SPI design.
> > > >
> > > > 1. Which backend assumptions should we design for?
> > > >
> > > > Within one Polaris server instance, do we assume entities, grants and
> > > > secrets all use the same database backend, or must we support mixing
> > > > backends across them?
> > > > Must every supported persistence implementation provide transactional
> > > > guarantees? Does the database itself need to provide transactions,
> or can
> > > > Polaris build that capability on top of simpler database operations?
> > > >
> > > > I'm happy to work with a homogeneous assumption if that's the
> intended
> > > > scope. Existing NoSQL illustrates the second question: it can publish
> > > > changes to multiple objects under one reference using a single
> > > conditional
> > > > pointer update.
> > > >
> > > > 2. How do we protect the data used by validation?
> > > >
> > > > Your storage-configuration and location-overlap examples make the
> problem
> > > > concrete. Consider these races:
> > > >
> > > > A table creation passes validation against the catalog's allowed
> > > locations,
> > > > then another request changes that configuration before the table is
> > > > committed.
> > > > Two table-creation requests each check for overlapping locations,
> both
> > > see
> > > > no overlap, and then create tables at conflicting locations.
> > > >
> > > > Checking only the record being written cannot cover these
> dependencies.
> > > The
> > > > first also depends on the catalog configuration. The second depends
> on
> > > the
> > > > absence of another overlapping table.
> > > >
> > > > Could the persistence layer protect these reads together with the
> writes,
> > > > using backend-internal versions/tokens or database isolation? Shared
> code
> > > > would still need to identify the required reads and checks. Could we
> keep
> > > > the mechanism for protecting them inside each backend?
> > > >
> > > > 3. Business rules are duplicated across backend implementations
> > > >
> > > > Today, the TreeMap
> > > > <
> > > >
> > >
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481
> > > > >
> > > > and JDBC
> > > > <
> > > >
> > >
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046
> > > > >
> > > > implementations each load a secret, check its principal, apply the
> > > > rotation/reset steps and write it back.
> > > >
> > > > Changing these business rules therefore requires keeping multiple
> > > > implementations in sync. A fix applied to one backend can be missed
> in
> > > > another, causing the same Polaris operation to behave differently
> > > depending
> > > > on the database.
> > > >
> > > > My proposal is to implement each business workflow once, against a
> common
> > > > set of storage operations with defined consistency and failure
> semantics.
> > > > The shared workflow should not need to know whether those operations
> are
> > > > implemented through JDBC transactions, NoSQL reference CAS or another
> > > > mechanism. Each backend would implement the same storage contract,
> > > keeping
> > > > its execution details internal. A new database backend could then
> reuse
> > > the
> > > > existing business workflows.
> > > >
> > > > Do these seem like reasonable starting points? We can then evaluate
> the
> > > SPI
> > > > against the original failure cases on both JDBC and existing NoSQL.
> > > >
> > > > -ej
> > > >
> > > > On Fri, Sep 18, 2026 at 12:10 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Side note: I just came across [5541], which shows a case where a
> failed
> > > > > request leaves persisted side effects.
> > > > >
> > > > > This can be seen as a coding mistake, of course. However, this is
> about
> > > > > executing one request as a single, atomic unit. This is something
> that
> > > > > probably needs foundational support and coding patterns in Polaris
> > > code.
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > > [5541] https://github.com/apache/polaris/pull/5541
> > > > >
> > > > > On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Here's a recap of the Community Sync discussion as I remember
> it. I'm
> > > > > sure
> > > > > > my recollection is not complete, so please add your thoughts to
> this
> > > > > thread.
> > > > > >
> > > > > > * Polaris traditionally relies on entity version number to ensure
> > > > > > consistency of changes.
> > > > > >
> > > > > > This works well for internal RBAC grants, which also have version
> > > > numbers
> > > > > > cross-referenced at the Persistence layer.
> > > > > >
> > > > > > I do not think this works in more general cases like validating
> > > > locations
> > > > > > wrt catalog-level storage configuration.
> > > > > >
> > > > > > Another difficult case involves location overlaps between
> concurrent
> > > > > table
> > > > > > creations.
> > > > > >
> > > > > > * We talked about how Persistence calls related to RDBMS
> transactions
> > > > (in
> > > > > > the JDBC persistence case).
> > > > > >
> > > > > > There were some concerns raised about whether we need to bind
> > > > Persistence
> > > > > > API to transactions explicitly.
> > > > > >
> > > > > > I think it is a valid concern. At the same time, the Persistence
> API
> > > > > > should be clear about consistency guarantees across all backend
> > > > > > implementations. I think this needs more attention now that the
> > > matter
> > > > of
> > > > > > milti-entioty changes became prominent in [5035].
> > > > > >
> > > > > > I personally believe that RDBMS-based Persistence in Polaris has
> to
> > > use
> > > > > > Serializable transaction isolation. Consequently, the core code
> needs
> > > > to
> > > > > > provide mechanisms to allow the Persistence implementation to
> > > properly
> > > > > > connect reads and writes from API requests to JDBC
> transactions...
> > > > > whether
> > > > > > we use "transation" as an explicit terms in the SPI or not.
> > > > > >
> > > > > > Note that currently, the same API request performs reads and
> writes
> > > in
> > > > > > _separate_ RDBMS transactions (even separate connections).
> > > > > >
> > > > > > * We did not discuss Persistence SPI implications in the call,
> but
> > > > > > currently each Persistence call is assumed to be one distinct and
> > > > atomic
> > > > > > change (please correct me if I'm wrong).
> > > > > >
> > > > > > Consequently, supporting new use cases involves introducing new
> SPI
> > > > > > methods with many parameters. This can be seen in PR [5035]. This
> > > tends
> > > > > to
> > > > > > overcomplicate the SPI.
> > > > > >
> > > > > > Ideally, I think the SPI should be simplified to deal with
> > > multi-entity
> > > > > > and single-entity changes in a coherent manner to avoid any
> ambiguity
> > > > > about
> > > > > > which method the caller should use.
> > > > > >
> > > > > > [5035] https://github.com/apache/polaris/pull/5035
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <
> > > dimas@apache.org
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Heads up: This discussion is on the Community Sync call agenda
> for
> > > > Sept
> > > > > >> 17.
> > > > > >>
> > > > > >> Interested parties, please plan to attend, if possible :)
> > > > > >>
> > > > > >> Cheers,
> > > > > >> Dmitri.
> > > > > >>
> > > > > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi JB,
> > > > > >>>
> > > > > >>> I've added an agenda item to the next community sync call for
> this
> > > > > (Sept
> > > > > >>> 17, 2026).
> > > > > >>>
> > > > > >>> Re: key points: did you mean a real GH discussion or dev email
> > > (this
> > > > > >>> thread)? Just double checking :) I'm fine with either approach.
> > > > > >>>
> > > > > >>> Cheers,
> > > > > >>> Dmitri.
> > > > > >>>
> > > > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <
> > > > jb@nanthrax.net>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Hi Dmitri,
> > > > > >>>>
> > > > > >>>> I agree on the need to align on these points, though I'm not
> > > > entirely
> > > > > >>>> sure a dedicated meeting is necessary. Let's start by using
> some
> > > > time
> > > > > >>>> during the next community meeting to discuss it.
> > > > > >>>>
> > > > > >>>> In the meantime, what if we outline the key points in a GitHub
> > > > > >>>> Discussion first? We can then use that as a reference during
> the
> > > > > >>>> meeting.
> > > > > >>>>
> > > > > >>>> Regards,
> > > > > >>>> JB
> > > > > >>>>
> > > > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <
> > > > dimas@apache.org
> > > > > >
> > > > > >>>> wrote:
> > > > > >>>> >
> > > > > >>>> > Hi All,
> > > > > >>>> >
> > > > > >>>> > My impression from this thread is that it might be time for
> a
> > > call
> > > > > to
> > > > > >>>> > discuss all the related issues and try to arrive at a shared
> > > > > >>>> implementation
> > > > > >>>> > plan.
> > > > > >>>> >
> > > > > >>>> > I know meetings are not ideal, but in this case it might be
> > > > > >>>> beneficial as a
> > > > > >>>> > means for achieving a common understanding of the set of
> > > problems
> > > > > and
> > > > > >>>> > priorities related to this thread. A dedicated metrics
> meeting
> > > > > worked
> > > > > >>>> well
> > > > > >>>> > from my POV.
> > > > > >>>> >
> > > > > >>>> > Allocating a time slot in the next community meeting might
> be an
> > > > > >>>> option,
> > > > > >>>> > although I think we might need the full hour in this case.
> > > > > >>>> >
> > > > > >>>> > WDYT?
> > > > > >>>> >
> > > > > >>>> > Thanks,
> > > > > >>>> > Dmitri.
> > > > > >>>> >
> > > > > >>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <
> > > > > >>>> prithvisivasankar@gmail.com>
> > > > > >>>> > wrote:
> > > > > >>>> >
> > > > > >>>> > > Hi all,
> > > > > >>>> > >
> > > > > >>>> > > I rewrote https://github.com/apache/polaris/pull/5035,
> > > > following
> > > > > >>>> the
> > > > > >>>> > > discussion to focus only on the SPI foundation.
> > > > > >>>> > >
> > > > > >>>> > > What is in the PR now:
> > > > > >>>> > >
> > > > > >>>> > > - A written manager-level consistency contract in
> > > > > >>>> > >
> site/content/in-dev/unreleased/persistence-consistency.md.
> > > > > >>>> > > - CAS-aware EntityMutation / GrantMutation records.
> > > > > >>>> > > - MetaStoreChangeSet and
> BasePersistence#commitChangeSet,
> > > > with
> > > > > >>>> backends
> > > > > >>>> > > opting in via supportsAtomicMixedCommit().
> > > > > >>>> > > - commitChangeSet implementations for the in-memory
> TreeMap
> > > > > >>>> backend and
> > > > > >>>> > > JDBC.
> > > > > >>>> > > - AtomicOperationMetaStoreManager and
> > > > > >>>> TransactionalMetaStoreManagerImpl
> > > > > >>>> > > grant/revoke paths now build one change set per
> operation,
> > > > > >>>> falling back
> > > > > >>>> > > to
> > > > > >>>> > > individual operations when the backend does not support
> > > mixed
> > > > > >>>> commits.
> > > > > >>>> > > - Tests for the change-set API and TreeMap atomic
> commits.
> > > > > >>>> > >
> > > > > >>>> > > I intentionally left out:
> > > > > >>>> > >
> > > > > >>>> > > - Entity deletes in MetaStoreChangeSet.
> > > > > >>>> > > - Refactoring createCatalog, dropEntity, and
> renameEntity
> > > to
> > > > > use
> > > > > >>>> change
> > > > > >>>> > > sets.
> > > > > >>>> > > - NoSQL support for commitChangeSet.
> > > > > >>>> > >
> > > > > >>>> > > I am treating this as Phase 1 so the contract and the SPI
> > > > > primitive
> > > > > >>>> can be
> > > > > >>>> > > reviewed before the larger operation migrations land.
> > > > > >>>> > >
> > > > > >>>> > > Could you please take a look? In particular, I would like
> to
> > > > know
> > > > > >>>> whether
> > > > > >>>> > > the contract captures the consensus so far, or if it
> needs to
> > > > > >>>> address retry
> > > > > >>>> > > / external-work coordination before we merge this
> foundation.
> > > > > >>>> > >
> > > > > >>>> > > Thanks,
> > > > > >>>> > > Prithvi S
> > > > > >>>> > >
> > > > > >>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <
> > > > > >>>> dimas@apache.org>
> > > > > >>>> > > wrote:
> > > > > >>>> > >
> > > > > >>>> > > > Hi Robert,
> > > > > >>>> > > >
> > > > > >>>> > > > I agree that the client-visible operation (e.g. REST API
> > > > > request)
> > > > > >>>> > > consists
> > > > > >>>> > > > of many distinct phases. The database / persistence
> changes
> > > > are
> > > > > >>>> just one
> > > > > >>>> > > of
> > > > > >>>> > > > these phases. STS (as an example) is another. My point
> about
> > > > > >>>> transactions
> > > > > >>>> > > > in JDBC Persistence applies to the former (database
> changes)
> > > > > >>>> phase - one
> > > > > >>>> > > > retry attempt there should ideally involve exactly one
> > > > > >>>> transaction.
> > > > > >>>> > > >
> > > > > >>>> > > > We certainly need logic inside Polaris Servers to to
> > > > coordinate
> > > > > >>>> requests
> > > > > >>>> > > to
> > > > > >>>> > > > various external systems depending on outcomes from
> previous
> > > > > >>>> request
> > > > > >>>> > > > processing phases.
> > > > > >>>> > > >
> > > > > >>>> > > > I also agree that validation based on database state
> should
> > > be
> > > > > >>>> repeated
> > > > > >>>> > > > from scratch if we retry the database changes.
> > > > > >>>> > > >
> > > > > >>>> > > > However, internal RBAC authorization naturally has to
> happen
> > > > > >>>> inside the
> > > > > >>>> > > > database update phase (read-check-write). IIRC, that is
> part
> > > > of
> > > > > >>>> the
> > > > > >>>> > > > implicit consistency guarantees, which were discussed
> during
> > > > > early
> > > > > >>>> > > project
> > > > > >>>> > > > months (but never got written down in full clarity,
> > > > > >>>> unfortunately). If
> > > > > >>>> > > the
> > > > > >>>> > > > Authorization check is inside that phase for internal
> RBAC,
> > > I
> > > > > >>>> suppose it
> > > > > >>>> > > > will be there for other Authorizers too and that can
> involve
> > > > > >>>> external
> > > > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly expect
> > > > > >>>> authorization to be
> > > > > >>>> > > > efficient, but I'm not sure how quick it can be in
> practice.
> > > > > >>>> There's
> > > > > >>>> > > > certainly potential for delays in some (perhaps
> infrequent)
> > > > > cases.
> > > > > >>>> > > >
> > > > > >>>> > > > Cheers,
> > > > > >>>> > > > Dmitri.
> > > > > >>>> > > >
> > > > > >>>> > > >
> > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <
> > > snazy@snazy.de>
> > > > > >>>> wrote:
> > > > > >>>> > > >
> > > > > >>>> > > > > Hi Dmitri,
> > > > > >>>> > > > >
> > > > > >>>> > > > > I agree that one logical operation needs a consistent
> view
> > > > of
> > > > > >>>> the whole
> > > > > >>>> > > > > backend state.
> > > > > >>>> > > > > What worries me is treating one long JDBC transaction
> as
> > > the
> > > > > >>>> boundary
> > > > > >>>> > > of
> > > > > >>>> > > > > that operation and then retrying the complete request
> if
> > > the
> > > > > >>>> commit
> > > > > >>>> > > > fails.
> > > > > >>>> > > > >
> > > > > >>>> > > > > The request can also write metadata to object storage
> or
> > > > call
> > > > > >>>> STS.
> > > > > >>>> > > > > Each system can leave us with an outcome whose
> certainty
> > > we
> > > > > >>>> cannot
> > > > > >>>> > > know.
> > > > > >>>> > > > > The database may have committed before the connection
> was
> > > > > lost,
> > > > > >>>> an
> > > > > >>>> > > > > object-store write may have succeeded before a
> timeout, or
> > > > STS
> > > > > >>>> may have
> > > > > >>>> > > > > issued credentials before its response was lost.
> > > > > >>>> > > > > A database rollback cannot undo any of those effects,
> and
> > > > > >>>> retrying the
> > > > > >>>> > > > > whole request may repeat them.
> > > > > >>>> > > > >
> > > > > >>>> > > > > So I think the contract has to separate the overall
> > > > operation
> > > > > >>>> from each
> > > > > >>>> > > > > attempt to commit it.
> > > > > >>>> > > > > Each database or backend commit attempt should stay
> short.
> > > > > >>>> > > > > When we retry, validation and authorization need to
> use
> > > the
> > > > > >>>> state for
> > > > > >>>> > > > that
> > > > > >>>> > > > > new attempt, and we need clear rules for which
> external
> > > work
> > > > > >>>> can be
> > > > > >>>> > > > reused,
> > > > > >>>> > > > > repeated, cleaned up, or reconciled after an uncertain
> > > > > outcome.
> > > > > >>>> > > > >
> > > > > >>>> > > > > Some contextual state could be useful for carrying
> that
> > > > state.
> > > > > >>>> > > > > But the JDBC connection and transaction should be an
> > > > > >>>> implementation
> > > > > >>>> > > > detail
> > > > > >>>> > > > > inside it, not the boundary of the complete operation.
> > > > > >>>> > > > >
> > > > > >>>> > > > > Cheers,
> > > > > >>>> > > > > Robert
> > > > > >>>> > > > >
> > > > > >>>> > > > >
> > > > > >>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <
> > > > > >>>> dimas@apache.org
> > > > > >>>> > > >
> > > > > >>>> > > > > wrote:
> > > > > >>>> > > > >
> > > > > >>>> > > > > > Hi All,
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > The idea of a backend-agnostic change-set primitive
> > > sounds
> > > > > >>>> good.
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > However, I am not sure it is sufficient for all the
> use
> > > > > cases
> > > > > >>>> we
> > > > > >>>> > > > touched
> > > > > >>>> > > > > > here. More specifically, the state read by
> validation
> > > code
> > > > > is
> > > > > >>>> not
> > > > > >>>> > > > > > necessarily reflected in the change set:
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > * Unchanged entities might be considered by
> validation,
> > > > but
> > > > > >>>> changed
> > > > > >>>> > > in
> > > > > >>>> > > > a
> > > > > >>>> > > > > > parallel request.
> > > > > >>>> > > > > > * Even for changed entities the state read by
> validation
> > > > may
> > > > > >>>> differ
> > > > > >>>> > > > from
> > > > > >>>> > > > > > the state being altered. Some validation code talks
> to
> > > the
> > > > > >>>> MetaStore
> > > > > >>>> > > > > > directly, outside of the data produced by the
> Resolver.
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > I do not think Polaris offers any explicit
> mechanisms
> > > > (ATM)
> > > > > >>>> to ensure
> > > > > >>>> > > > > > consistency between these reads and subsequent
> writes.
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > As far as JDBC goes, running an overarching
> Transaction
> > > > > >>>> across all
> > > > > >>>> > > > reads
> > > > > >>>> > > > > > and writes in the same request, at the SERIALIZABLE
> > > > > isolation
> > > > > >>>> level
> > > > > >>>> > > in
> > > > > >>>> > > > > the
> > > > > >>>> > > > > > backing RDBMS could solve the problem, I think.
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > Indeed, such a transaction might be long to
> accommodate
> > > > > calls
> > > > > >>>> made by
> > > > > >>>> > > > > > Polaris to external storage, etc. However, is that a
> > > > > problem?
> > > > > >>>> I think
> > > > > >>>> > > > the
> > > > > >>>> > > > > > alternative is for the JDBC Persistence impl. to
> > > > "manually"
> > > > > >>>> track all
> > > > > >>>> > > > > reads
> > > > > >>>> > > > > > under the same request and redo them in the small
> > > > > transaction
> > > > > >>>> that
> > > > > >>>> > > > > persists
> > > > > >>>> > > > > > the writes. This will also add RDBMS overhead and
> > > require
> > > > > >>>> complex
> > > > > >>>> > > code
> > > > > >>>> > > > in
> > > > > >>>> > > > > > Polaris to handle the data properly.
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > Tracking the request-wide transaction can be done
> only
> > > for
> > > > > >>>> JDBC
> > > > > >>>> > > without
> > > > > >>>> > > > > > leaking "transaction" concepts to the NoSQL
> > > Persistence, I
> > > > > >>>> think.
> > > > > >>>> > > NoSQL
> > > > > >>>> > > > > > will use other mechanisms to ensure read/write
> > > > consistency.
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > Connecting to Robert's email (a parallel branch in
> this
> > > > > >>>> discussion),
> > > > > >>>> > > > I'd
> > > > > >>>> > > > > > like to propose this approach:
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > * Each request establishes a "Data Context"
> > > > > >>>> > > > > > - In JDBC the Data Context corresponds to a JDBC
> > > > > Connection
> > > > > >>>> + Tx
> > > > > >>>> > > > > > - In NoSQL the Data Context tracks one or more
> > > reference
> > > > > >>>> hashes
> > > > > >>>> > > > > > * All Persistence access in the same request goes
> > > through
> > > > > the
> > > > > >>>> same
> > > > > >>>> > > Data
> > > > > >>>> > > > > > Context
> > > > > >>>> > > > > > * All Persistence changes are committed once at the
> end
> > > of
> > > > > the
> > > > > >>>> > > request
> > > > > >>>> > > > > > - Not all changes have to be globally atomic. For
> > > > example,
> > > > > >>>> changes
> > > > > >>>> > > in
> > > > > >>>> > > > > > Catalogs A and B do not have to be atomic with
> respect
> > > to
> > > > > >>>> each other.
> > > > > >>>> > > > We
> > > > > >>>> > > > > > can go deeper into this later. This is relevant to
> > > NoSQL.
> > > > > >>>> > > > > > - Transactional backends like JDBC can, of course,
> > > > choose
> > > > > >>>> to make
> > > > > >>>> > > all
> > > > > >>>> > > > > > changes globally atomic.
> > > > > >>>> > > > > > * A commit can fail in two main ways:
> > > > > >>>> > > > > > - A retriable failure like an optimistic lock
> error or
> > > > Tx
> > > > > >>>> > > > > serializability
> > > > > >>>> > > > > > error
> > > > > >>>> > > > > > - A non-triable logical error (e.g. entity not
> found)
> > > > > >>>> > > > > > * On a retriable error the whole request is
> re-attempted
> > > > (as
> > > > > >>>> if
> > > > > >>>> > > > > resubmitted
> > > > > >>>> > > > > > by a client) a few times (configurable timeout).
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > WDYT?
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > Cheers,
> > > > > >>>> > > > > > Dmitri.
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste
> Onofré <
> > > > > >>>> > > jb@nanthrax.net
> > > > > >>>> > > > >
> > > > > >>>> > > > > > wrote:
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > > Hi all
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > I think Robert has a good point.
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > If the atomicity guarantee lives only in
> > > > BasePersistence,
> > > > > >>>> then the
> > > > > >>>> > > > > > > manager contract can't tell a caller whether the
> state
> > > > it
> > > > > >>>> read for
> > > > > >>>> > > > > > > validation/authorization/credential-vending is the
> > > same
> > > > > >>>> state that
> > > > > >>>> > > > > > > eventually commits. That's the actual bug class
> behind
> > > > the
> > > > > >>>> JDBC
> > > > > >>>> > > > > > > symptoms (not any single operation being
> non-atomic,
> > > but
> > > > > the
> > > > > >>>> > > contract
> > > > > >>>> > > > > > > being silent about it. Every operation-specific
> fix
> > > > (like
> > > > > >>>> #4939,
> > > > > >>>> > > > #5035
> > > > > >>>> > > > > > > or #5095) re-answers this question locally and it
> > > keeps
> > > > > >>>> recurring.
> > > > > >>>> > > So
> > > > > >>>> > > > > > > the deliverable should start with a written
> > > consistency
> > > > > >>>> contract at
> > > > > >>>> > > > > > > the manager level.
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > I'm not sure migrating everyone to
> > > > > >>>> > > TransactionalMetaStoreManagerImpl
> > > > > >>>> > > > > > > is a good idea. It would tie the logical change
> set
> > > to a
> > > > > DB
> > > > > >>>> > > > > > > transaction spanning the REST request. It means:
> > > > > >>>> > > > > > > - it holds a durable transaction open across slow
> > > > external
> > > > > >>>> work
> > > > > >>>> > > > > > > (credential vending, OPA, Ranger, ...)
> > > > > >>>> > > > > > > - it doesn't map to NoSQL
> > > > > >>>> > > > > > > - It wraps single-row updates in
> > > runWiithinTransaction,
> > > > > >>>> which is a
> > > > > >>>> > > > > > overhead
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > So, I think the transactional manager isn't a
> portable
> > > > > >>>> target. It's
> > > > > >>>> > > > > > > "only" one backend's strategy.
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > I think Privthi's approach is right. A
> > > backend-agnostic
> > > > > >>>> change-set
> > > > > >>>> > > > > > > primitive with a documented fallback is the
> correct
> > > > shape.
> > > > > >>>> There is
> > > > > >>>> > > > > > > one caveat: without carrying the original entitiy
> (not
> > > > > just
> > > > > >>>> the new
> > > > > >>>> > > > > > > one) the change set can't express optimistic
> > > > concurrency.
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > I propose the following multi-steps approach:
> > > > > >>>> > > > > > > 1. We write the manager-level consistency
> contract as
> > > > > >>>> Robert asked.
> > > > > >>>> > > > It
> > > > > >>>> > > > > > > should include the explicit statement that a
> logical
> > > > > change
> > > > > >>>> set is
> > > > > >>>> > > > not
> > > > > >>>> > > > > > > a request-scope DB transaction.
> > > > > >>>> > > > > > > 2. We make Compare And Swap
> (optimistic-concurrency
> > > > > pattern)
> > > > > >>>> > > baseline
> > > > > >>>> > > > > > > first-class in EntityMutation before merging the
> SPI
> > > > > >>>> > > > > > > 3. Then we refactor
> > > > createCatalog/dropEntity/renameEntity.
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > Thoughts?
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > Regards
> > > > > >>>> > > > > > > JB
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <
> > > > > >>>> snazy@snazy.de>
> > > > > >>>> > > wrote:
> > > > > >>>> > > > > > > >
> > > > > >>>> > > > > > > > Hi all,
> > > > > >>>> > > > > > > >
> > > > > >>>> > > > > > > > Yufei's clarification about where the atomicity
> > > > > guarantee
> > > > > >>>> is
> > > > > >>>> > > > defined
> > > > > >>>> > > > > > > seems
> > > > > >>>> > > > > > > > important.
> > > > > >>>> > > > > > > > If it is a property of the lower-level
> > > BasePersistence
> > > > > >>>> contract
> > > > > >>>> > > > > rather
> > > > > >>>> > > > > > > than
> > > > > >>>> > > > > > > > the general PolarisMetaStoreManager contract,
> the
> > > > > general
> > > > > >>>> > > contract
> > > > > >>>> > > > > does
> > > > > >>>> > > > > > > not
> > > > > >>>> > > > > > > > tell callers whether the state used for
> validation,
> > > > > >>>> > > authorization,
> > > > > >>>> > > > or
> > > > > >>>> > > > > > > > credential vending is consistent with the change
> > > that
> > > > > >>>> eventually
> > > > > >>>> > > > > > commits.
> > > > > >>>> > > > > > > >
> > > > > >>>> > > > > > > > The current PRs suggest that operation-specific
> > > > > >>>> multi-object
> > > > > >>>> > > > methods
> > > > > >>>> > > > > > can
> > > > > >>>> > > > > > > > fix individual cases, while leaving the same
> > > contract
> > > > > >>>> question to
> > > > > >>>> > > > > recur
> > > > > >>>> > > > > > > for
> > > > > >>>> > > > > > > > each new case.
> > > > > >>>> > > > > > > >
> > > > > >>>> > > > > > > > I would also avoid defining a logical change
> set as
> > > a
> > > > > >>>> database
> > > > > >>>> > > > > > > transaction
> > > > > >>>> > > > > > > > around an entire REST request.
> > > > > >>>> > > > > > > > That would tie the contract to the backend and
> > > > database,
> > > > > >>>> and
> > > > > >>>> > > could
> > > > > >>>> > > > > keep
> > > > > >>>> > > > > > > the
> > > > > >>>> > > > > > > > durable attempt open across slow external work.
> > > > > >>>> > > > > > > >
> > > > > >>>> > > > > > > > So is the choice really between the two current
> > > > manager
> > > > > >>>> > > > > > implementations,
> > > > > >>>> > > > > > > or
> > > > > >>>> > > > > > > > do we first need to revisit the boundary and
> > > > guarantees
> > > > > >>>> exposed
> > > > > >>>> > > to
> > > > > >>>> > > > > > their
> > > > > >>>> > > > > > > > callers?
> > > > > >>>> > > > > > > >
> > > > > >>>> > > > > > > > Cheers,
> > > > > >>>> > > > > > > > Robert
> > > > > >>>> > > > > > > >
> > > > > >>>> > > > > > > >
> > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri
> > > Bourlatchkov <
> > > > > >>>> > > > > dimas@apache.org
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > > wrote:
> > > > > >>>> > > > > > > >
> > > > > >>>> > > > > > > > > Hi Yufei,
> > > > > >>>> > > > > > > > >
> > > > > >>>> > > > > > > > > Thanks for the link. I stand corrected. The
> "one
> > > > > atomic
> > > > > >>>> change
> > > > > >>>> > > > per
> > > > > >>>> > > > > > > method"
> > > > > >>>> > > > > > > > > contract as defined in javadoc does apply
> only to
> > > > > >>>> > > BasePersistence
> > > > > >>>> > > > > > > > > and AtomicOperationMetaStoreManager (which
> > > delegates
> > > > > to
> > > > > >>>> > > > > > > BasePersistence).
> > > > > >>>> > > > > > > > >
> > > > > >>>> > > > > > > > > Note that all non-test Persistence
> implementations
> > > > in
> > > > > >>>> the
> > > > > >>>> > > Polaris
> > > > > >>>> > > > > > > codebase
> > > > > >>>> > > > > > > > > extend those classes (which is probably why I
> was
> > > > > >>>> confused
> > > > > >>>> > > about
> > > > > >>>> > > > > > > atomicity
> > > > > >>>> > > > > > > > > expectations).
> > > > > >>>> > > > > > > > >
> > > > > >>>> > > > > > > > > However, this creates a gap in the
> Persistence SPI
> > > > > >>>> > > specification.
> > > > > >>>> > > > > If
> > > > > >>>> > > > > > > > > other PolarisMetaStoreManager implementations
> do
> > > not
> > > > > >>>> have to
> > > > > >>>> > > > comply
> > > > > >>>> > > > > > > with
> > > > > >>>> > > > > > > > > this principle, it will create a conceptual
> > > > difficulty
> > > > > >>>> at call
> > > > > >>>> > > > > sites.
> > > > > >>>> > > > > > > How
> > > > > >>>> > > > > > > > > can PolarisMetaStoreManager callers reason
> about
> > > > > >>>> consistency
> > > > > >>>> > > and
> > > > > >>>> > > > > > > durability
> > > > > >>>> > > > > > > > > behaviours in general?
> > > > > >>>> > > > > > > > >
> > > > > >>>> > > > > > > > > I believe we need to address that as part of
> this
> > > > > >>>> discussion.
> > > > > >>>> > > > > > > > >
> > > > > >>>> > > > > > > > > Cheers,
> > > > > >>>> > > > > > > > > Dmitri.
> > > > > >>>> > > > > > > > >
> > > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <
> > > > > >>>> flyrain000@gmail.com
> > > > > >>>> > > >
> > > > > >>>> > > > > > wrote:
> > > > > >>>> > > > > > > > >
> > > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined
> with
> > > the
> > > > > >>>> idea that
> > > > > >>>> > > one
> > > > > >>>> > > > > > > method
> > > > > >>>> > > > > > > > > call
> > > > > >>>> > > > > > > > > > means one atomic change.
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > > > If the "MetaStore SPI" refers to the
> interface
> > > > > >>>> > > > > > > PolarisMetaStoreManager, I
> > > > > >>>> > > > > > > > > > don't think we've ever state each method to
> be
> > > > > >>>> atomic. We did
> > > > > >>>> > > > > > clarify
> > > > > >>>> > > > > > > > > > atomicity[1] in the interface
> BasePersistence
> > > > > though.
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > > > 1.
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > >
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > >
> > > > > >>>> > > > >
> > > > > >>>> > > >
> > > > > >>>> > >
> > > > > >>>>
> > > > >
> > > >
> > >
> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > > > Yufei
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri
> > > > > Bourlatchkov <
> > > > > >>>> > > > > > > dimas@apache.org>
> > > > > >>>> > > > > > > > > > wrote:
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > > > > Hi Yufei,
> > > > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > I agree that JDBC transactions must be
> handled
> > > > > more
> > > > > >>>> > > > explicitly.
> > > > > >>>> > > > > > > > > However,
> > > > > >>>> > > > > > > > > > > I'm not sure that simply moving to
> > > > > >>>> > > > > > > TransactionalMetaStoreManagerImpl is
> > > > > >>>> > > > > > > > > > > sufficient.
> > > > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined
> with
> > > the
> > > > > >>>> idea that
> > > > > >>>> > > one
> > > > > >>>> > > > > > > method
> > > > > >>>> > > > > > > > > call
> > > > > >>>> > > > > > > > > > > means one atomic change [1]. The
> > > "transactional"
> > > > > >>>> MetaStore
> > > > > >>>> > > > > impl.
> > > > > >>>> > > > > > is
> > > > > >>>> > > > > > > > > but a
> > > > > >>>> > > > > > > > > > > sub-case of that. It cannot alter the
> > > high-level
> > > > > >>>> contract.
> > > > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > We could add SPI methods having multiple
> > > object
> > > > > >>>> parameters
> > > > > >>>> > > to
> > > > > >>>> > > > > > > represent
> > > > > >>>> > > > > > > > > > > grouped changes, but I am not sure it
> will be
> > > a
> > > > > >>>> sound
> > > > > >>>> > > design.
> > > > > >>>> > > > > > This
> > > > > >>>> > > > > > > will
> > > > > >>>> > > > > > > > > > > bloat the interface surfaces and require
> extra
> > > > > >>>> impl. effort
> > > > > >>>> > > > for
> > > > > >>>> > > > > > > each
> > > > > >>>> > > > > > > > > > > backend type. More importantly, adding
> > > multi-arg
> > > > > >>>> change
> > > > > >>>> > > > methods
> > > > > >>>> > > > > > > still
> > > > > >>>> > > > > > > > > > won't
> > > > > >>>> > > > > > > > > > > address the problem of reads being
> consistent
> > > > with
> > > > > >>>> writes,
> > > > > >>>> > > > > > because
> > > > > >>>> > > > > > > each
> > > > > >>>> > > > > > > > > > > method call will still be independent
> > > regarding
> > > > > the
> > > > > >>>> data
> > > > > >>>> > > > stored
> > > > > >>>> > > > > > in
> > > > > >>>> > > > > > > the
> > > > > >>>> > > > > > > > > > > database.
> > > > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > I tend to think we need to introduce a
> "change
> > > > > set"
> > > > > >>>> or
> > > > > >>>> > > > "atomic
> > > > > >>>> > > > > > > batch"
> > > > > >>>> > > > > > > > > > > concept to core Persistence and associate
> each
> > > > > REST
> > > > > >>>> API
> > > > > >>>> > > > request
> > > > > >>>> > > > > > > with
> > > > > >>>> > > > > > > > > one
> > > > > >>>> > > > > > > > > > > such change set, which will be committed
> (or
> > > > > rolled
> > > > > >>>> back)
> > > > > >>>> > > at
> > > > > >>>> > > > > the
> > > > > >>>> > > > > > > end of
> > > > > >>>> > > > > > > > > > the
> > > > > >>>> > > > > > > > > > > request. I believe Ayush mentioned a
> similar
> > > > > >>>> concept in PR
> > > > > >>>> > > > 4939
> > > > > >>>> > > > > > > [2]. In
> > > > > >>>> > > > > > > > > > > JDBC each change set will naturally be
> > > > associated
> > > > > >>>> with an
> > > > > >>>> > > > RDBMS
> > > > > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each
> atomic
> > > > > >>>> change set
> > > > > >>>> > > > will
> > > > > >>>> > > > > be
> > > > > >>>> > > > > > > > > > > associated with one CAS operation on the
> > > > > underlying
> > > > > >>>> > > database.
> > > > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > [1]
> > > > > >>>> > > > > > >
> > > > > >>>>
> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
> > > > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > [2]
> > > > > >>>> > > > > > >
> > > > > >>>>
> > > https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > > > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > Cheers,
> > > > > >>>> > > > > > > > > > > Dmitri.
> > > > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
> > > > > >>>> > > > flyrain000@gmail.com
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > > wrote:
> > > > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > Thanks for raising this, Dmitri.
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > These are valid concerns, and they were
> > > > already
> > > > > >>>> > > recognized
> > > > > >>>> > > > > when
> > > > > >>>> > > > > > > we
> > > > > >>>> > > > > > > > > > > > introduced JDBC persistence to Polaris.
> At
> > > > that
> > > > > >>>> time, we
> > > > > >>>> > > > > chose
> > > > > >>>> > > > > > > to use
> > > > > >>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the
> JDBC
> > > > due
> > > > > >>>> to the
> > > > > >>>> > > > > > > simplicity. I
> > > > > >>>> > > > > > > > > > > think
> > > > > >>>> > > > > > > > > > > > most of the issues mentioned here can
> > > already
> > > > be
> > > > > >>>> > > addressed
> > > > > >>>> > > > by
> > > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl.
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > For example, rename is already wrapped
> in a
> > > > > >>>> transaction
> > > > > >>>> > > in
> > > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1].
> > > > > Similarly,
> > > > > >>>> catalog
> > > > > >>>> > > > > > > creation,
> > > > > >>>> > > > > > > > > > which
> > > > > >>>> > > > > > > > > > > > involves reading and creating multiple
> > > > objects,
> > > > > >>>> is also
> > > > > >>>> > > > > > executed
> > > > > >>>> > > > > > > > > > within a
> > > > > >>>> > > > > > > > > > > > transaction [2].
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > I see two possible directions:
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > 1.
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > Modify
> AtomicOperationMetaStoreManager
> > > > > >>>> together with
> > > > > >>>> > > the
> > > > > >>>> > > > > > > > > persistence
> > > > > >>>> > > > > > > > > > > > backends (such as JDBC) to provide
> the
> > > > > required
> > > > > >>>> > > > > consistency
> > > > > >>>> > > > > > > > > > guarantees
> > > > > >>>> > > > > > > > > > > > for
> > > > > >>>> > > > > > > > > > > > specific operations, similar to what
> > > > > >>>> > > > > > > > > > TransactionalMetaStoreManagerImpl
> > > > > >>>> > > > > > > > > > > > does.
> > > > > >>>> > > > > > > > > > > > 2.
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > Migrate the persistence backends
> (such as
> > > > > >>>> JDBC) to use
> > > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl
> > > directly.
> > > > > We
> > > > > >>>> may
> > > > > >>>> > > have
> > > > > >>>> > > > to
> > > > > >>>> > > > > > > deal
> > > > > >>>> > > > > > > > > with
> > > > > >>>> > > > > > > > > > > > transactional semantic mismatches
> across
> > > > > >>>> different
> > > > > >>>> > > > > > persistence
> > > > > >>>> > > > > > > > > > > backends.
> > > > > >>>> > > > > > > > > > > > For example, we would likely avoid
> using
> > > > > JDBC's
> > > > > >>>> > > > > > > > > > `runWithinTransaction`
> > > > > >>>> > > > > > > > > > > > for
> > > > > >>>> > > > > > > > > > > > single row updates, which adds
> additional
> > > > > >>>> overhead and
> > > > > >>>> > > > > > > complexity
> > > > > >>>> > > > > > > > > > > > without
> > > > > >>>> > > > > > > > > > > > benefits.
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > References:
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > 1.
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > >
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > >
> > > > > >>>> > > > >
> > > > > >>>> > > >
> > > > > >>>> > >
> > > > > >>>>
> > > > >
> > > >
> > >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > > > > >>>> > > > > > > > > > > > 2.
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > >
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > >
> > > > > >>>> > > > >
> > > > > >>>> > > >
> > > > > >>>> > >
> > > > > >>>>
> > > > >
> > > >
> > >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > Yufei
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri
> > > > > >>>> Bourlatchkov <
> > > > > >>>> > > > > > > > > dimas@apache.org>
> > > > > >>>> > > > > > > > > > > > wrote:
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > Hi all,
> > > > > >>>> > > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > Ayush and Prithvi recently
> contributed a
> > > > > couple
> > > > > >>>> of
> > > > > >>>> > > > > > interesting
> > > > > >>>> > > > > > > PRs:
> > > > > >>>> > > > > > > > > > > > > [4939], [5035].
> > > > > >>>> > > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > It looks like people are starting to
> > > > encounter
> > > > > >>>> > > > consistency
> > > > > >>>> > > > > > > issues
> > > > > >>>> > > > > > > > > in
> > > > > >>>> > > > > > > > > > > > > JDBC persistence.
> > > > > >>>> > > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight into
> the
> > > > > >>>> underlying
> > > > > >>>> > > > > issues.
> > > > > >>>> > > > > > > They
> > > > > >>>> > > > > > > > > > offer
> > > > > >>>> > > > > > > > > > > > > incremental fixes that can work.
> However,
> > > I
> > > > > >>>> believe it
> > > > > >>>> > > is
> > > > > >>>> > > > > > time
> > > > > >>>> > > > > > > for
> > > > > >>>> > > > > > > > > > the
> > > > > >>>> > > > > > > > > > > > > Polaris community to review and
> improve
> > > this
> > > > > >>>> area of
> > > > > >>>> > > the
> > > > > >>>> > > > > > > codebase
> > > > > >>>> > > > > > > > > > > > > holistically.
> > > > > >>>> > > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > By this, I mean finding a solution
> that
> > > can
> > > > be
> > > > > >>>> applied
> > > > > >>>> > > to
> > > > > >>>> > > > > all
> > > > > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC,
> > > > NoSQL)
> > > > > >>>> and
> > > > > >>>> > > > addresses
> > > > > >>>> > > > > > > these
> > > > > >>>> > > > > > > > > > > > > aspects:
> > > > > >>>> > > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent
> > > > changes
> > > > > >>>> where
> > > > > >>>> > > the
> > > > > >>>> > > > > > > service
> > > > > >>>> > > > > > > > > > reads
> > > > > >>>> > > > > > > > > > > > > and validates current catalog state,
> > > then
> > > > > >>>> commits a
> > > > > >>>> > > > > change
> > > > > >>>> > > > > > > (e.g.
> > > > > >>>> > > > > > > > > > > > > name clashes during renames).
> > > > > >>>> > > > > > > > > > > > > * Supporting consistent but
> independent
> > > > > changes
> > > > > >>>> to RBAC
> > > > > >>>> > > > > > grants
> > > > > >>>> > > > > > > and
> > > > > >>>> > > > > > > > > > > > > MetaStore entities. This
> independence is
> > > > > >>>> needed to
> > > > > >>>> > > > > support
> > > > > >>>> > > > > > > > > > > > > external authorizers like OPA and
> > > Ranger.
> > > > > >>>> > > > > > > > > > > > > * Supporting atomic changes across
> > > multiple
> > > > > >>>> similar
> > > > > >>>> > > > > entities.
> > > > > >>>> > > > > > > > > > > > > * Supporting authorization-based
> filtering
> > > > of
> > > > > >>>> list
> > > > > >>>> > > > > operations
> > > > > >>>> > > > > > > (cf.
> > > > > >>>> > > > > > > > > > > > > [4831]).
> > > > > >>>> > > > > > > > > > > > > * Supporting credential-vending
> decisions
> > > > that
> > > > > >>>> are
> > > > > >>>> > > rooted
> > > > > >>>> > > > > in
> > > > > >>>> > > > > > > the
> > > > > >>>> > > > > > > > > > > > > exact state of the catalog.
> > > > > >>>> > > > > > > > > > > > > * Supporting server-side retries for
> > > > transient
> > > > > >>>> > > > persistence
> > > > > >>>> > > > > > > failures
> > > > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability
> > > failures).
> > > > > >>>> > > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > Please share your comments and ideas.
> > > > > >>>> > > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > [4831]
> > > > > >>>> https://github.com/apache/polaris/pull/4831
> > > > > >>>> > > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > [4939]
> > > > > >>>> https://github.com/apache/polaris/pull/4939
> > > > > >>>> > > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > [5035]
> > > > > >>>> https://github.com/apache/polaris/pull/5035
> > > > > >>>> > > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > Thanks,
> > > > > >>>> > > > > > > > > > > > > Dmitri
> > > > > >>>> > > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > >
> > > > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > >
> > > > > >>>> > > > > > > > >
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > >
> > > > > >>>> > > > >
> > > > > >>>> > > >
> > > > > >>>> > >
> > > > > >>>>
> > > > > >>>
> > > > >
> > > >
> > >
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Dennis
I agree with you. I think "multi object conditional commits" rather
than a request-scoped transaction is aligned with what Robert and I
argued: commit attempts stay short, it maps to both JDBC and NoSQL. An
explicit read set checked at commit time stays compatible with
in-memory caching (I don't think a serializable only approach does
not).
I believe it is also a small step forward from what we already have.
The multi-table commit path already does a multi-entity CAS through
updateEntitiesPropertiesIfNotChanged. We should extend it.
I have some concerns about using PolarisResolutionManifest as is:
1. As Dmitri mentioned, the location overlap check does not go through
the manifest at all. So the commit "contract" needs predicate checks
that the backend should re-evaluate at commit time.
2. The manifest is not an "open book" today.
getPassthroughResolvedPath creates a single user Resolver on every
call and does not record the version it returned. I don't see how to
store the version easily today.
3. In your example the catalog is not pinned, but validating a table
location against the allowed locations depends on the catalog storage
configuration, so for a create it would have to be. I would prefer
recording versions on read, or pinning by default with an explicit
opt-out.
4. To simplify, I would define the read set as a small type in the
Persistence SPI, which the manifest produces, rather than making the
manifest itself the SPI.
I would propose to define the read set in Persistence SPI terms,
independent from the resolver. and also include predicate (location
overlap, name conflict) in the contract.
I think it's pretty close to what Dmitri is proposing (the
"Persistence Session "iidea).
Regards
JB
On Thu, Oct 1, 2026 at 2:41 AM Dennis Huo <dh...@apache.org> wrote:
>
> Agree we should formulate in terms of Persistence SPIs first.
>
> IIRC we talked about this in one of the community syncs but I forgot to
> bring it here -- the PolarisResolutionManifest in concept already
> represents *exactly* the set of "which elements were engaged during request
> processing".
>
> Adherence may have drifted, but at least early on, we made sure all the
> IcebergCatalog logic did *not* get to do new direct reads from the
> persistence layer, but instead needs to "read" from the *authorized* set of
> entities contained in a PolarisResolutionManifest (or declared as a
> "passthroughPath" == allow reading fresh from DB but had to pre-register it
> for authz purposes).
>
> It's important to keep this unified anyways because it really serves two
> things:
>
> 1. Declares the set of entities whose state you expect to resolve/snapshot
> up-front in the operation-handling, that the IcebergCatalog logic then
> operates on later; also explicitly declare which ones require fetching
> *fresh* versions mid-operation (addPassthroughPath)
> 2. Enforces that RBAC authz is performed against that pre-declared set of
> entities -- if later processign were allowed to access arbitrary other
> entities to make handler decisions, then how did we "prove" that the
> authorization engine actually allowed the request to access those arbitrary
> other entities? Even if it's just for internal-processing purposes, the
> very fact that something got accessed means there's a route to accidental
> leakage of unauthorized data.
>
> So in a way, solid/provable authz and solid/provable consistency go
> hand-in-hand.
>
> I think PolarisResolutionManifest in its current form is probably a bit
> opaque and doesn't 100% convey the semantics we want quite yet, but it
> seems like the right SPI starting point to extend or at least evolve from.
> Basically, aside from addPath and addPassthroughPath, there could be a
> notion of annotating a given path as being one of the "dependent states" to
> carry over into the later commit. Conceptually:
>
> // Prepare operation processing
> resolutionManifest.prefetch(rootCatalog, mustBeUnchangedAtCommitTime =
> false); // For this operation let's pretend we don't care if the root
> catalog changes before we commit the change to the tables
> resolutionManifest.prefetch(parentNamespace, mustBeUnchangedAtCommitTime =
> false); // For this operation let's pretend we don't care if the parent
> namespace changes before we commit the change to the tables
> resolutionManifest.prefetch(table1, mustBeUnchangedAtCommitTime = true);
> // Assume multi-table commit - table1 will have changes and also must not
> have already changed at commit time
> resolutionManifest.prefetch(table2, mustBeUnchangedAtCommitTime = true);
> // Same for table2
>
> // .... Do all the Iceberg processing work
>
> // When we commit, we use the resolutionManifest as the ledger of things we
> used that must still be at their unchanged version at time of commit
> metastore.doConditionalBatchOmmit(List.of(table1Updated, table2Updated),
> resolutionManifest.getSetOfUnchangedEntityRequirements());
>
> Overall this fits the optimistic-concurrency theme of Iceberg and Polaris
> in general, and would *intentionally* not try to express a full-fledged
> "Transaction" session in which you do a bunch of reads/writes/reads/writes
> on a single entity as if you're operating in an uncommitted block on a
> RDBMS, instead opting for the middle-ground of "multi-object conditional
> commits".
>
> On Tue, Sep 22, 2026 at 4:10 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi EJ,
> >
> > > 1. Which backend assumptions should we design for? [...]
> >
> > I know I used the term "transaction" a lot during the call, but the real
> > challenge is about establishing an SPI with clear contracts that could be
> > implemented by JDBC and NoSQL persistence using the database features
> > natural to each model.
> >
> > I am fairly confident that a JDBC impl. is possible with Serializable
> > transaction. We should be able to solve all the problems raised in this
> > thread. However, that will make in-memory caching impossible because the
> > database needs to see all the reads (and writes) in a Tx to be able to
> > ensure serializable guarantees. If a cache prevents a read from hitting the
> > database, that read (e.g. storage config) will not be a factor in
> > serializability.
> >
> > This connects to my earlier question about the in-memory cache and JDBC
> > (title: Usefulness of InMemoryEntityCache)
> >
> > Are different approaches possible with JDBC? Probably. What is involved?
> > This depends on the SPI contract, I think.
> >
> > > 2. How do we protect the data used by validation? [...]
> >
> > The point mentioned by Robert previously IIRC, is that we need to ground
> > all runtime decisions to a particular state of the catalog.
> >
> > If at commit time the state of the catalog in the database shifts from what
> > was used during validation, the request has to restart from scratch.
> >
> > Current JDBC Persistence does this only for relationships between entities
> > and grants and between versions of the same entity.
> >
> > Relationships between table locations and storage config are not actualized
> > for the purpose of state consistency tracking.
> >
> > From my POV we need to begin by formulating this Persistence state in the
> > Persistence SPI terms. Then, think how we can implement that for JDBC.
> >
> > NoSQL Persistence already has a per-catalog state (not granular), so wiring
> > the new SPI to NoSQL should not be too hard, I hope.
> >
> > The state in the SPI does not have to be global to the catalog (as in
> > NoSQL), but it needs to be able to express the idea of which elements were
> > engaged during request processing so serve as input to Persistence impl.
> > for cosistency guarantees. If Persistence caches data in memory (for
> > example), it will have to do so in a way that does not violate consistency
> > expectations.
> >
> > From my POV we could begin by defining a "Persistence Session" concept.
> > Each request is 1:1 with a session. All reads and writes go through the
> > session. The next step would be the JDBC impl., which probably needs some
> > brainstorming... but I wonder if we could reach a consensus at a high level
> > first.
> >
> > WDYT?
> >
> > Thanks,
> > Dmitri.
> >
> > On Tue, Sep 22, 2026 at 5:55 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> > > Thanks for the recap, Dmitri. I'd like to follow up on the three points I
> > > raised during the sync. We didn't have much time to discuss them, so it
> > > would help to confirm these before choosing an SPI design.
> > >
> > > 1. Which backend assumptions should we design for?
> > >
> > > Within one Polaris server instance, do we assume entities, grants and
> > > secrets all use the same database backend, or must we support mixing
> > > backends across them?
> > > Must every supported persistence implementation provide transactional
> > > guarantees? Does the database itself need to provide transactions, or can
> > > Polaris build that capability on top of simpler database operations?
> > >
> > > I'm happy to work with a homogeneous assumption if that's the intended
> > > scope. Existing NoSQL illustrates the second question: it can publish
> > > changes to multiple objects under one reference using a single
> > conditional
> > > pointer update.
> > >
> > > 2. How do we protect the data used by validation?
> > >
> > > Your storage-configuration and location-overlap examples make the problem
> > > concrete. Consider these races:
> > >
> > > A table creation passes validation against the catalog's allowed
> > locations,
> > > then another request changes that configuration before the table is
> > > committed.
> > > Two table-creation requests each check for overlapping locations, both
> > see
> > > no overlap, and then create tables at conflicting locations.
> > >
> > > Checking only the record being written cannot cover these dependencies.
> > The
> > > first also depends on the catalog configuration. The second depends on
> > the
> > > absence of another overlapping table.
> > >
> > > Could the persistence layer protect these reads together with the writes,
> > > using backend-internal versions/tokens or database isolation? Shared code
> > > would still need to identify the required reads and checks. Could we keep
> > > the mechanism for protecting them inside each backend?
> > >
> > > 3. Business rules are duplicated across backend implementations
> > >
> > > Today, the TreeMap
> > > <
> > >
> > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481
> > > >
> > > and JDBC
> > > <
> > >
> > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046
> > > >
> > > implementations each load a secret, check its principal, apply the
> > > rotation/reset steps and write it back.
> > >
> > > Changing these business rules therefore requires keeping multiple
> > > implementations in sync. A fix applied to one backend can be missed in
> > > another, causing the same Polaris operation to behave differently
> > depending
> > > on the database.
> > >
> > > My proposal is to implement each business workflow once, against a common
> > > set of storage operations with defined consistency and failure semantics.
> > > The shared workflow should not need to know whether those operations are
> > > implemented through JDBC transactions, NoSQL reference CAS or another
> > > mechanism. Each backend would implement the same storage contract,
> > keeping
> > > its execution details internal. A new database backend could then reuse
> > the
> > > existing business workflows.
> > >
> > > Do these seem like reasonable starting points? We can then evaluate the
> > SPI
> > > against the original failure cases on both JDBC and existing NoSQL.
> > >
> > > -ej
> > >
> > > On Fri, Sep 18, 2026 at 12:10 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Side note: I just came across [5541], which shows a case where a failed
> > > > request leaves persisted side effects.
> > > >
> > > > This can be seen as a coding mistake, of course. However, this is about
> > > > executing one request as a single, atomic unit. This is something that
> > > > probably needs foundational support and coding patterns in Polaris
> > code.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > > [5541] https://github.com/apache/polaris/pull/5541
> > > >
> > > > On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov <di...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Here's a recap of the Community Sync discussion as I remember it. I'm
> > > > sure
> > > > > my recollection is not complete, so please add your thoughts to this
> > > > thread.
> > > > >
> > > > > * Polaris traditionally relies on entity version number to ensure
> > > > > consistency of changes.
> > > > >
> > > > > This works well for internal RBAC grants, which also have version
> > > numbers
> > > > > cross-referenced at the Persistence layer.
> > > > >
> > > > > I do not think this works in more general cases like validating
> > > locations
> > > > > wrt catalog-level storage configuration.
> > > > >
> > > > > Another difficult case involves location overlaps between concurrent
> > > > table
> > > > > creations.
> > > > >
> > > > > * We talked about how Persistence calls related to RDBMS transactions
> > > (in
> > > > > the JDBC persistence case).
> > > > >
> > > > > There were some concerns raised about whether we need to bind
> > > Persistence
> > > > > API to transactions explicitly.
> > > > >
> > > > > I think it is a valid concern. At the same time, the Persistence API
> > > > > should be clear about consistency guarantees across all backend
> > > > > implementations. I think this needs more attention now that the
> > matter
> > > of
> > > > > milti-entioty changes became prominent in [5035].
> > > > >
> > > > > I personally believe that RDBMS-based Persistence in Polaris has to
> > use
> > > > > Serializable transaction isolation. Consequently, the core code needs
> > > to
> > > > > provide mechanisms to allow the Persistence implementation to
> > properly
> > > > > connect reads and writes from API requests to JDBC transactions...
> > > > whether
> > > > > we use "transation" as an explicit terms in the SPI or not.
> > > > >
> > > > > Note that currently, the same API request performs reads and writes
> > in
> > > > > _separate_ RDBMS transactions (even separate connections).
> > > > >
> > > > > * We did not discuss Persistence SPI implications in the call, but
> > > > > currently each Persistence call is assumed to be one distinct and
> > > atomic
> > > > > change (please correct me if I'm wrong).
> > > > >
> > > > > Consequently, supporting new use cases involves introducing new SPI
> > > > > methods with many parameters. This can be seen in PR [5035]. This
> > tends
> > > > to
> > > > > overcomplicate the SPI.
> > > > >
> > > > > Ideally, I think the SPI should be simplified to deal with
> > multi-entity
> > > > > and single-entity changes in a coherent manner to avoid any ambiguity
> > > > about
> > > > > which method the caller should use.
> > > > >
> > > > > [5035] https://github.com/apache/polaris/pull/5035
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <
> > dimas@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > >> Heads up: This discussion is on the Community Sync call agenda for
> > > Sept
> > > > >> 17.
> > > > >>
> > > > >> Interested parties, please plan to attend, if possible :)
> > > > >>
> > > > >> Cheers,
> > > > >> Dmitri.
> > > > >>
> > > > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi JB,
> > > > >>>
> > > > >>> I've added an agenda item to the next community sync call for this
> > > > (Sept
> > > > >>> 17, 2026).
> > > > >>>
> > > > >>> Re: key points: did you mean a real GH discussion or dev email
> > (this
> > > > >>> thread)? Just double checking :) I'm fine with either approach.
> > > > >>>
> > > > >>> Cheers,
> > > > >>> Dmitri.
> > > > >>>
> > > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <
> > > jb@nanthrax.net>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Hi Dmitri,
> > > > >>>>
> > > > >>>> I agree on the need to align on these points, though I'm not
> > > entirely
> > > > >>>> sure a dedicated meeting is necessary. Let's start by using some
> > > time
> > > > >>>> during the next community meeting to discuss it.
> > > > >>>>
> > > > >>>> In the meantime, what if we outline the key points in a GitHub
> > > > >>>> Discussion first? We can then use that as a reference during the
> > > > >>>> meeting.
> > > > >>>>
> > > > >>>> Regards,
> > > > >>>> JB
> > > > >>>>
> > > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <
> > > dimas@apache.org
> > > > >
> > > > >>>> wrote:
> > > > >>>> >
> > > > >>>> > Hi All,
> > > > >>>> >
> > > > >>>> > My impression from this thread is that it might be time for a
> > call
> > > > to
> > > > >>>> > discuss all the related issues and try to arrive at a shared
> > > > >>>> implementation
> > > > >>>> > plan.
> > > > >>>> >
> > > > >>>> > I know meetings are not ideal, but in this case it might be
> > > > >>>> beneficial as a
> > > > >>>> > means for achieving a common understanding of the set of
> > problems
> > > > and
> > > > >>>> > priorities related to this thread. A dedicated metrics meeting
> > > > worked
> > > > >>>> well
> > > > >>>> > from my POV.
> > > > >>>> >
> > > > >>>> > Allocating a time slot in the next community meeting might be an
> > > > >>>> option,
> > > > >>>> > although I think we might need the full hour in this case.
> > > > >>>> >
> > > > >>>> > WDYT?
> > > > >>>> >
> > > > >>>> > Thanks,
> > > > >>>> > Dmitri.
> > > > >>>> >
> > > > >>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <
> > > > >>>> prithvisivasankar@gmail.com>
> > > > >>>> > wrote:
> > > > >>>> >
> > > > >>>> > > Hi all,
> > > > >>>> > >
> > > > >>>> > > I rewrote https://github.com/apache/polaris/pull/5035,
> > > following
> > > > >>>> the
> > > > >>>> > > discussion to focus only on the SPI foundation.
> > > > >>>> > >
> > > > >>>> > > What is in the PR now:
> > > > >>>> > >
> > > > >>>> > > - A written manager-level consistency contract in
> > > > >>>> > > site/content/in-dev/unreleased/persistence-consistency.md.
> > > > >>>> > > - CAS-aware EntityMutation / GrantMutation records.
> > > > >>>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet,
> > > with
> > > > >>>> backends
> > > > >>>> > > opting in via supportsAtomicMixedCommit().
> > > > >>>> > > - commitChangeSet implementations for the in-memory TreeMap
> > > > >>>> backend and
> > > > >>>> > > JDBC.
> > > > >>>> > > - AtomicOperationMetaStoreManager and
> > > > >>>> TransactionalMetaStoreManagerImpl
> > > > >>>> > > grant/revoke paths now build one change set per operation,
> > > > >>>> falling back
> > > > >>>> > > to
> > > > >>>> > > individual operations when the backend does not support
> > mixed
> > > > >>>> commits.
> > > > >>>> > > - Tests for the change-set API and TreeMap atomic commits.
> > > > >>>> > >
> > > > >>>> > > I intentionally left out:
> > > > >>>> > >
> > > > >>>> > > - Entity deletes in MetaStoreChangeSet.
> > > > >>>> > > - Refactoring createCatalog, dropEntity, and renameEntity
> > to
> > > > use
> > > > >>>> change
> > > > >>>> > > sets.
> > > > >>>> > > - NoSQL support for commitChangeSet.
> > > > >>>> > >
> > > > >>>> > > I am treating this as Phase 1 so the contract and the SPI
> > > > primitive
> > > > >>>> can be
> > > > >>>> > > reviewed before the larger operation migrations land.
> > > > >>>> > >
> > > > >>>> > > Could you please take a look? In particular, I would like to
> > > know
> > > > >>>> whether
> > > > >>>> > > the contract captures the consensus so far, or if it needs to
> > > > >>>> address retry
> > > > >>>> > > / external-work coordination before we merge this foundation.
> > > > >>>> > >
> > > > >>>> > > Thanks,
> > > > >>>> > > Prithvi S
> > > > >>>> > >
> > > > >>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <
> > > > >>>> dimas@apache.org>
> > > > >>>> > > wrote:
> > > > >>>> > >
> > > > >>>> > > > Hi Robert,
> > > > >>>> > > >
> > > > >>>> > > > I agree that the client-visible operation (e.g. REST API
> > > > request)
> > > > >>>> > > consists
> > > > >>>> > > > of many distinct phases. The database / persistence changes
> > > are
> > > > >>>> just one
> > > > >>>> > > of
> > > > >>>> > > > these phases. STS (as an example) is another. My point about
> > > > >>>> transactions
> > > > >>>> > > > in JDBC Persistence applies to the former (database changes)
> > > > >>>> phase - one
> > > > >>>> > > > retry attempt there should ideally involve exactly one
> > > > >>>> transaction.
> > > > >>>> > > >
> > > > >>>> > > > We certainly need logic inside Polaris Servers to to
> > > coordinate
> > > > >>>> requests
> > > > >>>> > > to
> > > > >>>> > > > various external systems depending on outcomes from previous
> > > > >>>> request
> > > > >>>> > > > processing phases.
> > > > >>>> > > >
> > > > >>>> > > > I also agree that validation based on database state should
> > be
> > > > >>>> repeated
> > > > >>>> > > > from scratch if we retry the database changes.
> > > > >>>> > > >
> > > > >>>> > > > However, internal RBAC authorization naturally has to happen
> > > > >>>> inside the
> > > > >>>> > > > database update phase (read-check-write). IIRC, that is part
> > > of
> > > > >>>> the
> > > > >>>> > > > implicit consistency guarantees, which were discussed during
> > > > early
> > > > >>>> > > project
> > > > >>>> > > > months (but never got written down in full clarity,
> > > > >>>> unfortunately). If
> > > > >>>> > > the
> > > > >>>> > > > Authorization check is inside that phase for internal RBAC,
> > I
> > > > >>>> suppose it
> > > > >>>> > > > will be there for other Authorizers too and that can involve
> > > > >>>> external
> > > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly expect
> > > > >>>> authorization to be
> > > > >>>> > > > efficient, but I'm not sure how quick it can be in practice.
> > > > >>>> There's
> > > > >>>> > > > certainly potential for delays in some (perhaps infrequent)
> > > > cases.
> > > > >>>> > > >
> > > > >>>> > > > Cheers,
> > > > >>>> > > > Dmitri.
> > > > >>>> > > >
> > > > >>>> > > >
> > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <
> > snazy@snazy.de>
> > > > >>>> wrote:
> > > > >>>> > > >
> > > > >>>> > > > > Hi Dmitri,
> > > > >>>> > > > >
> > > > >>>> > > > > I agree that one logical operation needs a consistent view
> > > of
> > > > >>>> the whole
> > > > >>>> > > > > backend state.
> > > > >>>> > > > > What worries me is treating one long JDBC transaction as
> > the
> > > > >>>> boundary
> > > > >>>> > > of
> > > > >>>> > > > > that operation and then retrying the complete request if
> > the
> > > > >>>> commit
> > > > >>>> > > > fails.
> > > > >>>> > > > >
> > > > >>>> > > > > The request can also write metadata to object storage or
> > > call
> > > > >>>> STS.
> > > > >>>> > > > > Each system can leave us with an outcome whose certainty
> > we
> > > > >>>> cannot
> > > > >>>> > > know.
> > > > >>>> > > > > The database may have committed before the connection was
> > > > lost,
> > > > >>>> an
> > > > >>>> > > > > object-store write may have succeeded before a timeout, or
> > > STS
> > > > >>>> may have
> > > > >>>> > > > > issued credentials before its response was lost.
> > > > >>>> > > > > A database rollback cannot undo any of those effects, and
> > > > >>>> retrying the
> > > > >>>> > > > > whole request may repeat them.
> > > > >>>> > > > >
> > > > >>>> > > > > So I think the contract has to separate the overall
> > > operation
> > > > >>>> from each
> > > > >>>> > > > > attempt to commit it.
> > > > >>>> > > > > Each database or backend commit attempt should stay short.
> > > > >>>> > > > > When we retry, validation and authorization need to use
> > the
> > > > >>>> state for
> > > > >>>> > > > that
> > > > >>>> > > > > new attempt, and we need clear rules for which external
> > work
> > > > >>>> can be
> > > > >>>> > > > reused,
> > > > >>>> > > > > repeated, cleaned up, or reconciled after an uncertain
> > > > outcome.
> > > > >>>> > > > >
> > > > >>>> > > > > Some contextual state could be useful for carrying that
> > > state.
> > > > >>>> > > > > But the JDBC connection and transaction should be an
> > > > >>>> implementation
> > > > >>>> > > > detail
> > > > >>>> > > > > inside it, not the boundary of the complete operation.
> > > > >>>> > > > >
> > > > >>>> > > > > Cheers,
> > > > >>>> > > > > Robert
> > > > >>>> > > > >
> > > > >>>> > > > >
> > > > >>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <
> > > > >>>> dimas@apache.org
> > > > >>>> > > >
> > > > >>>> > > > > wrote:
> > > > >>>> > > > >
> > > > >>>> > > > > > Hi All,
> > > > >>>> > > > > >
> > > > >>>> > > > > > The idea of a backend-agnostic change-set primitive
> > sounds
> > > > >>>> good.
> > > > >>>> > > > > >
> > > > >>>> > > > > > However, I am not sure it is sufficient for all the use
> > > > cases
> > > > >>>> we
> > > > >>>> > > > touched
> > > > >>>> > > > > > here. More specifically, the state read by validation
> > code
> > > > is
> > > > >>>> not
> > > > >>>> > > > > > necessarily reflected in the change set:
> > > > >>>> > > > > >
> > > > >>>> > > > > > * Unchanged entities might be considered by validation,
> > > but
> > > > >>>> changed
> > > > >>>> > > in
> > > > >>>> > > > a
> > > > >>>> > > > > > parallel request.
> > > > >>>> > > > > > * Even for changed entities the state read by validation
> > > may
> > > > >>>> differ
> > > > >>>> > > > from
> > > > >>>> > > > > > the state being altered. Some validation code talks to
> > the
> > > > >>>> MetaStore
> > > > >>>> > > > > > directly, outside of the data produced by the Resolver.
> > > > >>>> > > > > >
> > > > >>>> > > > > > I do not think Polaris offers any explicit mechanisms
> > > (ATM)
> > > > >>>> to ensure
> > > > >>>> > > > > > consistency between these reads and subsequent writes.
> > > > >>>> > > > > >
> > > > >>>> > > > > > As far as JDBC goes, running an overarching Transaction
> > > > >>>> across all
> > > > >>>> > > > reads
> > > > >>>> > > > > > and writes in the same request, at the SERIALIZABLE
> > > > isolation
> > > > >>>> level
> > > > >>>> > > in
> > > > >>>> > > > > the
> > > > >>>> > > > > > backing RDBMS could solve the problem, I think.
> > > > >>>> > > > > >
> > > > >>>> > > > > > Indeed, such a transaction might be long to accommodate
> > > > calls
> > > > >>>> made by
> > > > >>>> > > > > > Polaris to external storage, etc. However, is that a
> > > > problem?
> > > > >>>> I think
> > > > >>>> > > > the
> > > > >>>> > > > > > alternative is for the JDBC Persistence impl. to
> > > "manually"
> > > > >>>> track all
> > > > >>>> > > > > reads
> > > > >>>> > > > > > under the same request and redo them in the small
> > > > transaction
> > > > >>>> that
> > > > >>>> > > > > persists
> > > > >>>> > > > > > the writes. This will also add RDBMS overhead and
> > require
> > > > >>>> complex
> > > > >>>> > > code
> > > > >>>> > > > in
> > > > >>>> > > > > > Polaris to handle the data properly.
> > > > >>>> > > > > >
> > > > >>>> > > > > > Tracking the request-wide transaction can be done only
> > for
> > > > >>>> JDBC
> > > > >>>> > > without
> > > > >>>> > > > > > leaking "transaction" concepts to the NoSQL
> > Persistence, I
> > > > >>>> think.
> > > > >>>> > > NoSQL
> > > > >>>> > > > > > will use other mechanisms to ensure read/write
> > > consistency.
> > > > >>>> > > > > >
> > > > >>>> > > > > > Connecting to Robert's email (a parallel branch in this
> > > > >>>> discussion),
> > > > >>>> > > > I'd
> > > > >>>> > > > > > like to propose this approach:
> > > > >>>> > > > > >
> > > > >>>> > > > > > * Each request establishes a "Data Context"
> > > > >>>> > > > > > - In JDBC the Data Context corresponds to a JDBC
> > > > Connection
> > > > >>>> + Tx
> > > > >>>> > > > > > - In NoSQL the Data Context tracks one or more
> > reference
> > > > >>>> hashes
> > > > >>>> > > > > > * All Persistence access in the same request goes
> > through
> > > > the
> > > > >>>> same
> > > > >>>> > > Data
> > > > >>>> > > > > > Context
> > > > >>>> > > > > > * All Persistence changes are committed once at the end
> > of
> > > > the
> > > > >>>> > > request
> > > > >>>> > > > > > - Not all changes have to be globally atomic. For
> > > example,
> > > > >>>> changes
> > > > >>>> > > in
> > > > >>>> > > > > > Catalogs A and B do not have to be atomic with respect
> > to
> > > > >>>> each other.
> > > > >>>> > > > We
> > > > >>>> > > > > > can go deeper into this later. This is relevant to
> > NoSQL.
> > > > >>>> > > > > > - Transactional backends like JDBC can, of course,
> > > choose
> > > > >>>> to make
> > > > >>>> > > all
> > > > >>>> > > > > > changes globally atomic.
> > > > >>>> > > > > > * A commit can fail in two main ways:
> > > > >>>> > > > > > - A retriable failure like an optimistic lock error or
> > > Tx
> > > > >>>> > > > > serializability
> > > > >>>> > > > > > error
> > > > >>>> > > > > > - A non-triable logical error (e.g. entity not found)
> > > > >>>> > > > > > * On a retriable error the whole request is re-attempted
> > > (as
> > > > >>>> if
> > > > >>>> > > > > resubmitted
> > > > >>>> > > > > > by a client) a few times (configurable timeout).
> > > > >>>> > > > > >
> > > > >>>> > > > > > WDYT?
> > > > >>>> > > > > >
> > > > >>>> > > > > > Cheers,
> > > > >>>> > > > > > Dmitri.
> > > > >>>> > > > > >
> > > > >>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré <
> > > > >>>> > > jb@nanthrax.net
> > > > >>>> > > > >
> > > > >>>> > > > > > wrote:
> > > > >>>> > > > > >
> > > > >>>> > > > > > > Hi all
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > I think Robert has a good point.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > If the atomicity guarantee lives only in
> > > BasePersistence,
> > > > >>>> then the
> > > > >>>> > > > > > > manager contract can't tell a caller whether the state
> > > it
> > > > >>>> read for
> > > > >>>> > > > > > > validation/authorization/credential-vending is the
> > same
> > > > >>>> state that
> > > > >>>> > > > > > > eventually commits. That's the actual bug class behind
> > > the
> > > > >>>> JDBC
> > > > >>>> > > > > > > symptoms (not any single operation being non-atomic,
> > but
> > > > the
> > > > >>>> > > contract
> > > > >>>> > > > > > > being silent about it. Every operation-specific fix
> > > (like
> > > > >>>> #4939,
> > > > >>>> > > > #5035
> > > > >>>> > > > > > > or #5095) re-answers this question locally and it
> > keeps
> > > > >>>> recurring.
> > > > >>>> > > So
> > > > >>>> > > > > > > the deliverable should start with a written
> > consistency
> > > > >>>> contract at
> > > > >>>> > > > > > > the manager level.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > I'm not sure migrating everyone to
> > > > >>>> > > TransactionalMetaStoreManagerImpl
> > > > >>>> > > > > > > is a good idea. It would tie the logical change set
> > to a
> > > > DB
> > > > >>>> > > > > > > transaction spanning the REST request. It means:
> > > > >>>> > > > > > > - it holds a durable transaction open across slow
> > > external
> > > > >>>> work
> > > > >>>> > > > > > > (credential vending, OPA, Ranger, ...)
> > > > >>>> > > > > > > - it doesn't map to NoSQL
> > > > >>>> > > > > > > - It wraps single-row updates in
> > runWiithinTransaction,
> > > > >>>> which is a
> > > > >>>> > > > > > overhead
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > So, I think the transactional manager isn't a portable
> > > > >>>> target. It's
> > > > >>>> > > > > > > "only" one backend's strategy.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > I think Privthi's approach is right. A
> > backend-agnostic
> > > > >>>> change-set
> > > > >>>> > > > > > > primitive with a documented fallback is the correct
> > > shape.
> > > > >>>> There is
> > > > >>>> > > > > > > one caveat: without carrying the original entitiy (not
> > > > just
> > > > >>>> the new
> > > > >>>> > > > > > > one) the change set can't express optimistic
> > > concurrency.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > I propose the following multi-steps approach:
> > > > >>>> > > > > > > 1. We write the manager-level consistency contract as
> > > > >>>> Robert asked.
> > > > >>>> > > > It
> > > > >>>> > > > > > > should include the explicit statement that a logical
> > > > change
> > > > >>>> set is
> > > > >>>> > > > not
> > > > >>>> > > > > > > a request-scope DB transaction.
> > > > >>>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency
> > > > pattern)
> > > > >>>> > > baseline
> > > > >>>> > > > > > > first-class in EntityMutation before merging the SPI
> > > > >>>> > > > > > > 3. Then we refactor
> > > createCatalog/dropEntity/renameEntity.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > Thoughts?
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > Regards
> > > > >>>> > > > > > > JB
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <
> > > > >>>> snazy@snazy.de>
> > > > >>>> > > wrote:
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > Hi all,
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > Yufei's clarification about where the atomicity
> > > > guarantee
> > > > >>>> is
> > > > >>>> > > > defined
> > > > >>>> > > > > > > seems
> > > > >>>> > > > > > > > important.
> > > > >>>> > > > > > > > If it is a property of the lower-level
> > BasePersistence
> > > > >>>> contract
> > > > >>>> > > > > rather
> > > > >>>> > > > > > > than
> > > > >>>> > > > > > > > the general PolarisMetaStoreManager contract, the
> > > > general
> > > > >>>> > > contract
> > > > >>>> > > > > does
> > > > >>>> > > > > > > not
> > > > >>>> > > > > > > > tell callers whether the state used for validation,
> > > > >>>> > > authorization,
> > > > >>>> > > > or
> > > > >>>> > > > > > > > credential vending is consistent with the change
> > that
> > > > >>>> eventually
> > > > >>>> > > > > > commits.
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > The current PRs suggest that operation-specific
> > > > >>>> multi-object
> > > > >>>> > > > methods
> > > > >>>> > > > > > can
> > > > >>>> > > > > > > > fix individual cases, while leaving the same
> > contract
> > > > >>>> question to
> > > > >>>> > > > > recur
> > > > >>>> > > > > > > for
> > > > >>>> > > > > > > > each new case.
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > I would also avoid defining a logical change set as
> > a
> > > > >>>> database
> > > > >>>> > > > > > > transaction
> > > > >>>> > > > > > > > around an entire REST request.
> > > > >>>> > > > > > > > That would tie the contract to the backend and
> > > database,
> > > > >>>> and
> > > > >>>> > > could
> > > > >>>> > > > > keep
> > > > >>>> > > > > > > the
> > > > >>>> > > > > > > > durable attempt open across slow external work.
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > So is the choice really between the two current
> > > manager
> > > > >>>> > > > > > implementations,
> > > > >>>> > > > > > > or
> > > > >>>> > > > > > > > do we first need to revisit the boundary and
> > > guarantees
> > > > >>>> exposed
> > > > >>>> > > to
> > > > >>>> > > > > > their
> > > > >>>> > > > > > > > callers?
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > Cheers,
> > > > >>>> > > > > > > > Robert
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri
> > Bourlatchkov <
> > > > >>>> > > > > dimas@apache.org
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > > wrote:
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > > Hi Yufei,
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Thanks for the link. I stand corrected. The "one
> > > > atomic
> > > > >>>> change
> > > > >>>> > > > per
> > > > >>>> > > > > > > method"
> > > > >>>> > > > > > > > > contract as defined in javadoc does apply only to
> > > > >>>> > > BasePersistence
> > > > >>>> > > > > > > > > and AtomicOperationMetaStoreManager (which
> > delegates
> > > > to
> > > > >>>> > > > > > > BasePersistence).
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Note that all non-test Persistence implementations
> > > in
> > > > >>>> the
> > > > >>>> > > Polaris
> > > > >>>> > > > > > > codebase
> > > > >>>> > > > > > > > > extend those classes (which is probably why I was
> > > > >>>> confused
> > > > >>>> > > about
> > > > >>>> > > > > > > atomicity
> > > > >>>> > > > > > > > > expectations).
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > However, this creates a gap in the Persistence SPI
> > > > >>>> > > specification.
> > > > >>>> > > > > If
> > > > >>>> > > > > > > > > other PolarisMetaStoreManager implementations do
> > not
> > > > >>>> have to
> > > > >>>> > > > comply
> > > > >>>> > > > > > > with
> > > > >>>> > > > > > > > > this principle, it will create a conceptual
> > > difficulty
> > > > >>>> at call
> > > > >>>> > > > > sites.
> > > > >>>> > > > > > > How
> > > > >>>> > > > > > > > > can PolarisMetaStoreManager callers reason about
> > > > >>>> consistency
> > > > >>>> > > and
> > > > >>>> > > > > > > durability
> > > > >>>> > > > > > > > > behaviours in general?
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > I believe we need to address that as part of this
> > > > >>>> discussion.
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Cheers,
> > > > >>>> > > > > > > > > Dmitri.
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <
> > > > >>>> flyrain000@gmail.com
> > > > >>>> > > >
> > > > >>>> > > > > > wrote:
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with
> > the
> > > > >>>> idea that
> > > > >>>> > > one
> > > > >>>> > > > > > > method
> > > > >>>> > > > > > > > > call
> > > > >>>> > > > > > > > > > means one atomic change.
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > If the "MetaStore SPI" refers to the interface
> > > > >>>> > > > > > > PolarisMetaStoreManager, I
> > > > >>>> > > > > > > > > > don't think we've ever state each method to be
> > > > >>>> atomic. We did
> > > > >>>> > > > > > clarify
> > > > >>>> > > > > > > > > > atomicity[1] in the interface BasePersistence
> > > > though.
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > 1.
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>>
> > > >
> > >
> > https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > Yufei
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri
> > > > Bourlatchkov <
> > > > >>>> > > > > > > dimas@apache.org>
> > > > >>>> > > > > > > > > > wrote:
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > > Hi Yufei,
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > I agree that JDBC transactions must be handled
> > > > more
> > > > >>>> > > > explicitly.
> > > > >>>> > > > > > > > > However,
> > > > >>>> > > > > > > > > > > I'm not sure that simply moving to
> > > > >>>> > > > > > > TransactionalMetaStoreManagerImpl is
> > > > >>>> > > > > > > > > > > sufficient.
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with
> > the
> > > > >>>> idea that
> > > > >>>> > > one
> > > > >>>> > > > > > > method
> > > > >>>> > > > > > > > > call
> > > > >>>> > > > > > > > > > > means one atomic change [1]. The
> > "transactional"
> > > > >>>> MetaStore
> > > > >>>> > > > > impl.
> > > > >>>> > > > > > is
> > > > >>>> > > > > > > > > but a
> > > > >>>> > > > > > > > > > > sub-case of that. It cannot alter the
> > high-level
> > > > >>>> contract.
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > We could add SPI methods having multiple
> > object
> > > > >>>> parameters
> > > > >>>> > > to
> > > > >>>> > > > > > > represent
> > > > >>>> > > > > > > > > > > grouped changes, but I am not sure it will be
> > a
> > > > >>>> sound
> > > > >>>> > > design.
> > > > >>>> > > > > > This
> > > > >>>> > > > > > > will
> > > > >>>> > > > > > > > > > > bloat the interface surfaces and require extra
> > > > >>>> impl. effort
> > > > >>>> > > > for
> > > > >>>> > > > > > > each
> > > > >>>> > > > > > > > > > > backend type. More importantly, adding
> > multi-arg
> > > > >>>> change
> > > > >>>> > > > methods
> > > > >>>> > > > > > > still
> > > > >>>> > > > > > > > > > won't
> > > > >>>> > > > > > > > > > > address the problem of reads being consistent
> > > with
> > > > >>>> writes,
> > > > >>>> > > > > > because
> > > > >>>> > > > > > > each
> > > > >>>> > > > > > > > > > > method call will still be independent
> > regarding
> > > > the
> > > > >>>> data
> > > > >>>> > > > stored
> > > > >>>> > > > > > in
> > > > >>>> > > > > > > the
> > > > >>>> > > > > > > > > > > database.
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > I tend to think we need to introduce a "change
> > > > set"
> > > > >>>> or
> > > > >>>> > > > "atomic
> > > > >>>> > > > > > > batch"
> > > > >>>> > > > > > > > > > > concept to core Persistence and associate each
> > > > REST
> > > > >>>> API
> > > > >>>> > > > request
> > > > >>>> > > > > > > with
> > > > >>>> > > > > > > > > one
> > > > >>>> > > > > > > > > > > such change set, which will be committed (or
> > > > rolled
> > > > >>>> back)
> > > > >>>> > > at
> > > > >>>> > > > > the
> > > > >>>> > > > > > > end of
> > > > >>>> > > > > > > > > > the
> > > > >>>> > > > > > > > > > > request. I believe Ayush mentioned a similar
> > > > >>>> concept in PR
> > > > >>>> > > > 4939
> > > > >>>> > > > > > > [2]. In
> > > > >>>> > > > > > > > > > > JDBC each change set will naturally be
> > > associated
> > > > >>>> with an
> > > > >>>> > > > RDBMS
> > > > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each atomic
> > > > >>>> change set
> > > > >>>> > > > will
> > > > >>>> > > > > be
> > > > >>>> > > > > > > > > > > associated with one CAS operation on the
> > > > underlying
> > > > >>>> > > database.
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > [1]
> > > > >>>> > > > > > >
> > > > >>>> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > [2]
> > > > >>>> > > > > > >
> > > > >>>>
> > https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > Cheers,
> > > > >>>> > > > > > > > > > > Dmitri.
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
> > > > >>>> > > > flyrain000@gmail.com
> > > > >>>> > > > > >
> > > > >>>> > > > > > > wrote:
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > Thanks for raising this, Dmitri.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > These are valid concerns, and they were
> > > already
> > > > >>>> > > recognized
> > > > >>>> > > > > when
> > > > >>>> > > > > > > we
> > > > >>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At
> > > that
> > > > >>>> time, we
> > > > >>>> > > > > chose
> > > > >>>> > > > > > > to use
> > > > >>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC
> > > due
> > > > >>>> to the
> > > > >>>> > > > > > > simplicity. I
> > > > >>>> > > > > > > > > > > think
> > > > >>>> > > > > > > > > > > > most of the issues mentioned here can
> > already
> > > be
> > > > >>>> > > addressed
> > > > >>>> > > > by
> > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > For example, rename is already wrapped in a
> > > > >>>> transaction
> > > > >>>> > > in
> > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1].
> > > > Similarly,
> > > > >>>> catalog
> > > > >>>> > > > > > > creation,
> > > > >>>> > > > > > > > > > which
> > > > >>>> > > > > > > > > > > > involves reading and creating multiple
> > > objects,
> > > > >>>> is also
> > > > >>>> > > > > > executed
> > > > >>>> > > > > > > > > > within a
> > > > >>>> > > > > > > > > > > > transaction [2].
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > I see two possible directions:
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > 1.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager
> > > > >>>> together with
> > > > >>>> > > the
> > > > >>>> > > > > > > > > persistence
> > > > >>>> > > > > > > > > > > > backends (such as JDBC) to provide the
> > > > required
> > > > >>>> > > > > consistency
> > > > >>>> > > > > > > > > > guarantees
> > > > >>>> > > > > > > > > > > > for
> > > > >>>> > > > > > > > > > > > specific operations, similar to what
> > > > >>>> > > > > > > > > > TransactionalMetaStoreManagerImpl
> > > > >>>> > > > > > > > > > > > does.
> > > > >>>> > > > > > > > > > > > 2.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > Migrate the persistence backends (such as
> > > > >>>> JDBC) to use
> > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl
> > directly.
> > > > We
> > > > >>>> may
> > > > >>>> > > have
> > > > >>>> > > > to
> > > > >>>> > > > > > > deal
> > > > >>>> > > > > > > > > with
> > > > >>>> > > > > > > > > > > > transactional semantic mismatches across
> > > > >>>> different
> > > > >>>> > > > > > persistence
> > > > >>>> > > > > > > > > > > backends.
> > > > >>>> > > > > > > > > > > > For example, we would likely avoid using
> > > > JDBC's
> > > > >>>> > > > > > > > > > `runWithinTransaction`
> > > > >>>> > > > > > > > > > > > for
> > > > >>>> > > > > > > > > > > > single row updates, which adds additional
> > > > >>>> overhead and
> > > > >>>> > > > > > > complexity
> > > > >>>> > > > > > > > > > > > without
> > > > >>>> > > > > > > > > > > > benefits.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > References:
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > 1.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>>
> > > >
> > >
> > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > > > >>>> > > > > > > > > > > > 2.
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>>
> > > >
> > >
> > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > Yufei
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri
> > > > >>>> Bourlatchkov <
> > > > >>>> > > > > > > > > dimas@apache.org>
> > > > >>>> > > > > > > > > > > > wrote:
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > Hi all,
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a
> > > > couple
> > > > >>>> of
> > > > >>>> > > > > > interesting
> > > > >>>> > > > > > > PRs:
> > > > >>>> > > > > > > > > > > > > [4939], [5035].
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > It looks like people are starting to
> > > encounter
> > > > >>>> > > > consistency
> > > > >>>> > > > > > > issues
> > > > >>>> > > > > > > > > in
> > > > >>>> > > > > > > > > > > > > JDBC persistence.
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight into the
> > > > >>>> underlying
> > > > >>>> > > > > issues.
> > > > >>>> > > > > > > They
> > > > >>>> > > > > > > > > > offer
> > > > >>>> > > > > > > > > > > > > incremental fixes that can work. However,
> > I
> > > > >>>> believe it
> > > > >>>> > > is
> > > > >>>> > > > > > time
> > > > >>>> > > > > > > for
> > > > >>>> > > > > > > > > > the
> > > > >>>> > > > > > > > > > > > > Polaris community to review and improve
> > this
> > > > >>>> area of
> > > > >>>> > > the
> > > > >>>> > > > > > > codebase
> > > > >>>> > > > > > > > > > > > > holistically.
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > By this, I mean finding a solution that
> > can
> > > be
> > > > >>>> applied
> > > > >>>> > > to
> > > > >>>> > > > > all
> > > > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC,
> > > NoSQL)
> > > > >>>> and
> > > > >>>> > > > addresses
> > > > >>>> > > > > > > these
> > > > >>>> > > > > > > > > > > > > aspects:
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent
> > > changes
> > > > >>>> where
> > > > >>>> > > the
> > > > >>>> > > > > > > service
> > > > >>>> > > > > > > > > > reads
> > > > >>>> > > > > > > > > > > > > and validates current catalog state,
> > then
> > > > >>>> commits a
> > > > >>>> > > > > change
> > > > >>>> > > > > > > (e.g.
> > > > >>>> > > > > > > > > > > > > name clashes during renames).
> > > > >>>> > > > > > > > > > > > > * Supporting consistent but independent
> > > > changes
> > > > >>>> to RBAC
> > > > >>>> > > > > > grants
> > > > >>>> > > > > > > and
> > > > >>>> > > > > > > > > > > > > MetaStore entities. This independence is
> > > > >>>> needed to
> > > > >>>> > > > > support
> > > > >>>> > > > > > > > > > > > > external authorizers like OPA and
> > Ranger.
> > > > >>>> > > > > > > > > > > > > * Supporting atomic changes across
> > multiple
> > > > >>>> similar
> > > > >>>> > > > > entities.
> > > > >>>> > > > > > > > > > > > > * Supporting authorization-based filtering
> > > of
> > > > >>>> list
> > > > >>>> > > > > operations
> > > > >>>> > > > > > > (cf.
> > > > >>>> > > > > > > > > > > > > [4831]).
> > > > >>>> > > > > > > > > > > > > * Supporting credential-vending decisions
> > > that
> > > > >>>> are
> > > > >>>> > > rooted
> > > > >>>> > > > > in
> > > > >>>> > > > > > > the
> > > > >>>> > > > > > > > > > > > > exact state of the catalog.
> > > > >>>> > > > > > > > > > > > > * Supporting server-side retries for
> > > transient
> > > > >>>> > > > persistence
> > > > >>>> > > > > > > failures
> > > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability
> > failures).
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > Please share your comments and ideas.
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > [4831]
> > > > >>>> https://github.com/apache/polaris/pull/4831
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > [4939]
> > > > >>>> https://github.com/apache/polaris/pull/4939
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > [5035]
> > > > >>>> https://github.com/apache/polaris/pull/5035
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > Thanks,
> > > > >>>> > > > > > > > > > > > > Dmitri
> > > > >>>> > > > > > > > > > > > >
> > > > >>>> > > > > > > > > > > >
> > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > >
> > > > >>>> > > >
> > > > >>>> > >
> > > > >>>>
> > > > >>>
> > > >
> > >
> >
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Dennis Huo <dh...@apache.org>.
Agree we should formulate in terms of Persistence SPIs first.
IIRC we talked about this in one of the community syncs but I forgot to
bring it here -- the PolarisResolutionManifest in concept already
represents *exactly* the set of "which elements were engaged during request
processing".
Adherence may have drifted, but at least early on, we made sure all the
IcebergCatalog logic did *not* get to do new direct reads from the
persistence layer, but instead needs to "read" from the *authorized* set of
entities contained in a PolarisResolutionManifest (or declared as a
"passthroughPath" == allow reading fresh from DB but had to pre-register it
for authz purposes).
It's important to keep this unified anyways because it really serves two
things:
1. Declares the set of entities whose state you expect to resolve/snapshot
up-front in the operation-handling, that the IcebergCatalog logic then
operates on later; also explicitly declare which ones require fetching
*fresh* versions mid-operation (addPassthroughPath)
2. Enforces that RBAC authz is performed against that pre-declared set of
entities -- if later processign were allowed to access arbitrary other
entities to make handler decisions, then how did we "prove" that the
authorization engine actually allowed the request to access those arbitrary
other entities? Even if it's just for internal-processing purposes, the
very fact that something got accessed means there's a route to accidental
leakage of unauthorized data.
So in a way, solid/provable authz and solid/provable consistency go
hand-in-hand.
I think PolarisResolutionManifest in its current form is probably a bit
opaque and doesn't 100% convey the semantics we want quite yet, but it
seems like the right SPI starting point to extend or at least evolve from.
Basically, aside from addPath and addPassthroughPath, there could be a
notion of annotating a given path as being one of the "dependent states" to
carry over into the later commit. Conceptually:
// Prepare operation processing
resolutionManifest.prefetch(rootCatalog, mustBeUnchangedAtCommitTime =
false); // For this operation let's pretend we don't care if the root
catalog changes before we commit the change to the tables
resolutionManifest.prefetch(parentNamespace, mustBeUnchangedAtCommitTime =
false); // For this operation let's pretend we don't care if the parent
namespace changes before we commit the change to the tables
resolutionManifest.prefetch(table1, mustBeUnchangedAtCommitTime = true);
// Assume multi-table commit - table1 will have changes and also must not
have already changed at commit time
resolutionManifest.prefetch(table2, mustBeUnchangedAtCommitTime = true);
// Same for table2
// .... Do all the Iceberg processing work
// When we commit, we use the resolutionManifest as the ledger of things we
used that must still be at their unchanged version at time of commit
metastore.doConditionalBatchOmmit(List.of(table1Updated, table2Updated),
resolutionManifest.getSetOfUnchangedEntityRequirements());
Overall this fits the optimistic-concurrency theme of Iceberg and Polaris
in general, and would *intentionally* not try to express a full-fledged
"Transaction" session in which you do a bunch of reads/writes/reads/writes
on a single entity as if you're operating in an uncommitted block on a
RDBMS, instead opting for the middle-ground of "multi-object conditional
commits".
On Tue, Sep 22, 2026 at 4:10 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi EJ,
>
> > 1. Which backend assumptions should we design for? [...]
>
> I know I used the term "transaction" a lot during the call, but the real
> challenge is about establishing an SPI with clear contracts that could be
> implemented by JDBC and NoSQL persistence using the database features
> natural to each model.
>
> I am fairly confident that a JDBC impl. is possible with Serializable
> transaction. We should be able to solve all the problems raised in this
> thread. However, that will make in-memory caching impossible because the
> database needs to see all the reads (and writes) in a Tx to be able to
> ensure serializable guarantees. If a cache prevents a read from hitting the
> database, that read (e.g. storage config) will not be a factor in
> serializability.
>
> This connects to my earlier question about the in-memory cache and JDBC
> (title: Usefulness of InMemoryEntityCache)
>
> Are different approaches possible with JDBC? Probably. What is involved?
> This depends on the SPI contract, I think.
>
> > 2. How do we protect the data used by validation? [...]
>
> The point mentioned by Robert previously IIRC, is that we need to ground
> all runtime decisions to a particular state of the catalog.
>
> If at commit time the state of the catalog in the database shifts from what
> was used during validation, the request has to restart from scratch.
>
> Current JDBC Persistence does this only for relationships between entities
> and grants and between versions of the same entity.
>
> Relationships between table locations and storage config are not actualized
> for the purpose of state consistency tracking.
>
> From my POV we need to begin by formulating this Persistence state in the
> Persistence SPI terms. Then, think how we can implement that for JDBC.
>
> NoSQL Persistence already has a per-catalog state (not granular), so wiring
> the new SPI to NoSQL should not be too hard, I hope.
>
> The state in the SPI does not have to be global to the catalog (as in
> NoSQL), but it needs to be able to express the idea of which elements were
> engaged during request processing so serve as input to Persistence impl.
> for cosistency guarantees. If Persistence caches data in memory (for
> example), it will have to do so in a way that does not violate consistency
> expectations.
>
> From my POV we could begin by defining a "Persistence Session" concept.
> Each request is 1:1 with a session. All reads and writes go through the
> session. The next step would be the JDBC impl., which probably needs some
> brainstorming... but I wonder if we could reach a consensus at a high level
> first.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> On Tue, Sep 22, 2026 at 5:55 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > Thanks for the recap, Dmitri. I'd like to follow up on the three points I
> > raised during the sync. We didn't have much time to discuss them, so it
> > would help to confirm these before choosing an SPI design.
> >
> > 1. Which backend assumptions should we design for?
> >
> > Within one Polaris server instance, do we assume entities, grants and
> > secrets all use the same database backend, or must we support mixing
> > backends across them?
> > Must every supported persistence implementation provide transactional
> > guarantees? Does the database itself need to provide transactions, or can
> > Polaris build that capability on top of simpler database operations?
> >
> > I'm happy to work with a homogeneous assumption if that's the intended
> > scope. Existing NoSQL illustrates the second question: it can publish
> > changes to multiple objects under one reference using a single
> conditional
> > pointer update.
> >
> > 2. How do we protect the data used by validation?
> >
> > Your storage-configuration and location-overlap examples make the problem
> > concrete. Consider these races:
> >
> > A table creation passes validation against the catalog's allowed
> locations,
> > then another request changes that configuration before the table is
> > committed.
> > Two table-creation requests each check for overlapping locations, both
> see
> > no overlap, and then create tables at conflicting locations.
> >
> > Checking only the record being written cannot cover these dependencies.
> The
> > first also depends on the catalog configuration. The second depends on
> the
> > absence of another overlapping table.
> >
> > Could the persistence layer protect these reads together with the writes,
> > using backend-internal versions/tokens or database isolation? Shared code
> > would still need to identify the required reads and checks. Could we keep
> > the mechanism for protecting them inside each backend?
> >
> > 3. Business rules are duplicated across backend implementations
> >
> > Today, the TreeMap
> > <
> >
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481
> > >
> > and JDBC
> > <
> >
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046
> > >
> > implementations each load a secret, check its principal, apply the
> > rotation/reset steps and write it back.
> >
> > Changing these business rules therefore requires keeping multiple
> > implementations in sync. A fix applied to one backend can be missed in
> > another, causing the same Polaris operation to behave differently
> depending
> > on the database.
> >
> > My proposal is to implement each business workflow once, against a common
> > set of storage operations with defined consistency and failure semantics.
> > The shared workflow should not need to know whether those operations are
> > implemented through JDBC transactions, NoSQL reference CAS or another
> > mechanism. Each backend would implement the same storage contract,
> keeping
> > its execution details internal. A new database backend could then reuse
> the
> > existing business workflows.
> >
> > Do these seem like reasonable starting points? We can then evaluate the
> SPI
> > against the original failure cases on both JDBC and existing NoSQL.
> >
> > -ej
> >
> > On Fri, Sep 18, 2026 at 12:10 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Side note: I just came across [5541], which shows a case where a failed
> > > request leaves persisted side effects.
> > >
> > > This can be seen as a coding mistake, of course. However, this is about
> > > executing one request as a single, atomic unit. This is something that
> > > probably needs foundational support and coding patterns in Polaris
> code.
> > >
> > > Cheers,
> > > Dmitri.
> > > [5541] https://github.com/apache/polaris/pull/5541
> > >
> > > On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Here's a recap of the Community Sync discussion as I remember it. I'm
> > > sure
> > > > my recollection is not complete, so please add your thoughts to this
> > > thread.
> > > >
> > > > * Polaris traditionally relies on entity version number to ensure
> > > > consistency of changes.
> > > >
> > > > This works well for internal RBAC grants, which also have version
> > numbers
> > > > cross-referenced at the Persistence layer.
> > > >
> > > > I do not think this works in more general cases like validating
> > locations
> > > > wrt catalog-level storage configuration.
> > > >
> > > > Another difficult case involves location overlaps between concurrent
> > > table
> > > > creations.
> > > >
> > > > * We talked about how Persistence calls related to RDBMS transactions
> > (in
> > > > the JDBC persistence case).
> > > >
> > > > There were some concerns raised about whether we need to bind
> > Persistence
> > > > API to transactions explicitly.
> > > >
> > > > I think it is a valid concern. At the same time, the Persistence API
> > > > should be clear about consistency guarantees across all backend
> > > > implementations. I think this needs more attention now that the
> matter
> > of
> > > > milti-entioty changes became prominent in [5035].
> > > >
> > > > I personally believe that RDBMS-based Persistence in Polaris has to
> use
> > > > Serializable transaction isolation. Consequently, the core code needs
> > to
> > > > provide mechanisms to allow the Persistence implementation to
> properly
> > > > connect reads and writes from API requests to JDBC transactions...
> > > whether
> > > > we use "transation" as an explicit terms in the SPI or not.
> > > >
> > > > Note that currently, the same API request performs reads and writes
> in
> > > > _separate_ RDBMS transactions (even separate connections).
> > > >
> > > > * We did not discuss Persistence SPI implications in the call, but
> > > > currently each Persistence call is assumed to be one distinct and
> > atomic
> > > > change (please correct me if I'm wrong).
> > > >
> > > > Consequently, supporting new use cases involves introducing new SPI
> > > > methods with many parameters. This can be seen in PR [5035]. This
> tends
> > > to
> > > > overcomplicate the SPI.
> > > >
> > > > Ideally, I think the SPI should be simplified to deal with
> multi-entity
> > > > and single-entity changes in a coherent manner to avoid any ambiguity
> > > about
> > > > which method the caller should use.
> > > >
> > > > [5035] https://github.com/apache/polaris/pull/5035
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <
> dimas@apache.org
> > >
> > > > wrote:
> > > >
> > > >> Heads up: This discussion is on the Community Sync call agenda for
> > Sept
> > > >> 17.
> > > >>
> > > >> Interested parties, please plan to attend, if possible :)
> > > >>
> > > >> Cheers,
> > > >> Dmitri.
> > > >>
> > > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > >> wrote:
> > > >>
> > > >>> Hi JB,
> > > >>>
> > > >>> I've added an agenda item to the next community sync call for this
> > > (Sept
> > > >>> 17, 2026).
> > > >>>
> > > >>> Re: key points: did you mean a real GH discussion or dev email
> (this
> > > >>> thread)? Just double checking :) I'm fine with either approach.
> > > >>>
> > > >>> Cheers,
> > > >>> Dmitri.
> > > >>>
> > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <
> > jb@nanthrax.net>
> > > >>> wrote:
> > > >>>
> > > >>>> Hi Dmitri,
> > > >>>>
> > > >>>> I agree on the need to align on these points, though I'm not
> > entirely
> > > >>>> sure a dedicated meeting is necessary. Let's start by using some
> > time
> > > >>>> during the next community meeting to discuss it.
> > > >>>>
> > > >>>> In the meantime, what if we outline the key points in a GitHub
> > > >>>> Discussion first? We can then use that as a reference during the
> > > >>>> meeting.
> > > >>>>
> > > >>>> Regards,
> > > >>>> JB
> > > >>>>
> > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <
> > dimas@apache.org
> > > >
> > > >>>> wrote:
> > > >>>> >
> > > >>>> > Hi All,
> > > >>>> >
> > > >>>> > My impression from this thread is that it might be time for a
> call
> > > to
> > > >>>> > discuss all the related issues and try to arrive at a shared
> > > >>>> implementation
> > > >>>> > plan.
> > > >>>> >
> > > >>>> > I know meetings are not ideal, but in this case it might be
> > > >>>> beneficial as a
> > > >>>> > means for achieving a common understanding of the set of
> problems
> > > and
> > > >>>> > priorities related to this thread. A dedicated metrics meeting
> > > worked
> > > >>>> well
> > > >>>> > from my POV.
> > > >>>> >
> > > >>>> > Allocating a time slot in the next community meeting might be an
> > > >>>> option,
> > > >>>> > although I think we might need the full hour in this case.
> > > >>>> >
> > > >>>> > WDYT?
> > > >>>> >
> > > >>>> > Thanks,
> > > >>>> > Dmitri.
> > > >>>> >
> > > >>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <
> > > >>>> prithvisivasankar@gmail.com>
> > > >>>> > wrote:
> > > >>>> >
> > > >>>> > > Hi all,
> > > >>>> > >
> > > >>>> > > I rewrote https://github.com/apache/polaris/pull/5035,
> > following
> > > >>>> the
> > > >>>> > > discussion to focus only on the SPI foundation.
> > > >>>> > >
> > > >>>> > > What is in the PR now:
> > > >>>> > >
> > > >>>> > > - A written manager-level consistency contract in
> > > >>>> > > site/content/in-dev/unreleased/persistence-consistency.md.
> > > >>>> > > - CAS-aware EntityMutation / GrantMutation records.
> > > >>>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet,
> > with
> > > >>>> backends
> > > >>>> > > opting in via supportsAtomicMixedCommit().
> > > >>>> > > - commitChangeSet implementations for the in-memory TreeMap
> > > >>>> backend and
> > > >>>> > > JDBC.
> > > >>>> > > - AtomicOperationMetaStoreManager and
> > > >>>> TransactionalMetaStoreManagerImpl
> > > >>>> > > grant/revoke paths now build one change set per operation,
> > > >>>> falling back
> > > >>>> > > to
> > > >>>> > > individual operations when the backend does not support
> mixed
> > > >>>> commits.
> > > >>>> > > - Tests for the change-set API and TreeMap atomic commits.
> > > >>>> > >
> > > >>>> > > I intentionally left out:
> > > >>>> > >
> > > >>>> > > - Entity deletes in MetaStoreChangeSet.
> > > >>>> > > - Refactoring createCatalog, dropEntity, and renameEntity
> to
> > > use
> > > >>>> change
> > > >>>> > > sets.
> > > >>>> > > - NoSQL support for commitChangeSet.
> > > >>>> > >
> > > >>>> > > I am treating this as Phase 1 so the contract and the SPI
> > > primitive
> > > >>>> can be
> > > >>>> > > reviewed before the larger operation migrations land.
> > > >>>> > >
> > > >>>> > > Could you please take a look? In particular, I would like to
> > know
> > > >>>> whether
> > > >>>> > > the contract captures the consensus so far, or if it needs to
> > > >>>> address retry
> > > >>>> > > / external-work coordination before we merge this foundation.
> > > >>>> > >
> > > >>>> > > Thanks,
> > > >>>> > > Prithvi S
> > > >>>> > >
> > > >>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <
> > > >>>> dimas@apache.org>
> > > >>>> > > wrote:
> > > >>>> > >
> > > >>>> > > > Hi Robert,
> > > >>>> > > >
> > > >>>> > > > I agree that the client-visible operation (e.g. REST API
> > > request)
> > > >>>> > > consists
> > > >>>> > > > of many distinct phases. The database / persistence changes
> > are
> > > >>>> just one
> > > >>>> > > of
> > > >>>> > > > these phases. STS (as an example) is another. My point about
> > > >>>> transactions
> > > >>>> > > > in JDBC Persistence applies to the former (database changes)
> > > >>>> phase - one
> > > >>>> > > > retry attempt there should ideally involve exactly one
> > > >>>> transaction.
> > > >>>> > > >
> > > >>>> > > > We certainly need logic inside Polaris Servers to to
> > coordinate
> > > >>>> requests
> > > >>>> > > to
> > > >>>> > > > various external systems depending on outcomes from previous
> > > >>>> request
> > > >>>> > > > processing phases.
> > > >>>> > > >
> > > >>>> > > > I also agree that validation based on database state should
> be
> > > >>>> repeated
> > > >>>> > > > from scratch if we retry the database changes.
> > > >>>> > > >
> > > >>>> > > > However, internal RBAC authorization naturally has to happen
> > > >>>> inside the
> > > >>>> > > > database update phase (read-check-write). IIRC, that is part
> > of
> > > >>>> the
> > > >>>> > > > implicit consistency guarantees, which were discussed during
> > > early
> > > >>>> > > project
> > > >>>> > > > months (but never got written down in full clarity,
> > > >>>> unfortunately). If
> > > >>>> > > the
> > > >>>> > > > Authorization check is inside that phase for internal RBAC,
> I
> > > >>>> suppose it
> > > >>>> > > > will be there for other Authorizers too and that can involve
> > > >>>> external
> > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly expect
> > > >>>> authorization to be
> > > >>>> > > > efficient, but I'm not sure how quick it can be in practice.
> > > >>>> There's
> > > >>>> > > > certainly potential for delays in some (perhaps infrequent)
> > > cases.
> > > >>>> > > >
> > > >>>> > > > Cheers,
> > > >>>> > > > Dmitri.
> > > >>>> > > >
> > > >>>> > > >
> > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <
> snazy@snazy.de>
> > > >>>> wrote:
> > > >>>> > > >
> > > >>>> > > > > Hi Dmitri,
> > > >>>> > > > >
> > > >>>> > > > > I agree that one logical operation needs a consistent view
> > of
> > > >>>> the whole
> > > >>>> > > > > backend state.
> > > >>>> > > > > What worries me is treating one long JDBC transaction as
> the
> > > >>>> boundary
> > > >>>> > > of
> > > >>>> > > > > that operation and then retrying the complete request if
> the
> > > >>>> commit
> > > >>>> > > > fails.
> > > >>>> > > > >
> > > >>>> > > > > The request can also write metadata to object storage or
> > call
> > > >>>> STS.
> > > >>>> > > > > Each system can leave us with an outcome whose certainty
> we
> > > >>>> cannot
> > > >>>> > > know.
> > > >>>> > > > > The database may have committed before the connection was
> > > lost,
> > > >>>> an
> > > >>>> > > > > object-store write may have succeeded before a timeout, or
> > STS
> > > >>>> may have
> > > >>>> > > > > issued credentials before its response was lost.
> > > >>>> > > > > A database rollback cannot undo any of those effects, and
> > > >>>> retrying the
> > > >>>> > > > > whole request may repeat them.
> > > >>>> > > > >
> > > >>>> > > > > So I think the contract has to separate the overall
> > operation
> > > >>>> from each
> > > >>>> > > > > attempt to commit it.
> > > >>>> > > > > Each database or backend commit attempt should stay short.
> > > >>>> > > > > When we retry, validation and authorization need to use
> the
> > > >>>> state for
> > > >>>> > > > that
> > > >>>> > > > > new attempt, and we need clear rules for which external
> work
> > > >>>> can be
> > > >>>> > > > reused,
> > > >>>> > > > > repeated, cleaned up, or reconciled after an uncertain
> > > outcome.
> > > >>>> > > > >
> > > >>>> > > > > Some contextual state could be useful for carrying that
> > state.
> > > >>>> > > > > But the JDBC connection and transaction should be an
> > > >>>> implementation
> > > >>>> > > > detail
> > > >>>> > > > > inside it, not the boundary of the complete operation.
> > > >>>> > > > >
> > > >>>> > > > > Cheers,
> > > >>>> > > > > Robert
> > > >>>> > > > >
> > > >>>> > > > >
> > > >>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <
> > > >>>> dimas@apache.org
> > > >>>> > > >
> > > >>>> > > > > wrote:
> > > >>>> > > > >
> > > >>>> > > > > > Hi All,
> > > >>>> > > > > >
> > > >>>> > > > > > The idea of a backend-agnostic change-set primitive
> sounds
> > > >>>> good.
> > > >>>> > > > > >
> > > >>>> > > > > > However, I am not sure it is sufficient for all the use
> > > cases
> > > >>>> we
> > > >>>> > > > touched
> > > >>>> > > > > > here. More specifically, the state read by validation
> code
> > > is
> > > >>>> not
> > > >>>> > > > > > necessarily reflected in the change set:
> > > >>>> > > > > >
> > > >>>> > > > > > * Unchanged entities might be considered by validation,
> > but
> > > >>>> changed
> > > >>>> > > in
> > > >>>> > > > a
> > > >>>> > > > > > parallel request.
> > > >>>> > > > > > * Even for changed entities the state read by validation
> > may
> > > >>>> differ
> > > >>>> > > > from
> > > >>>> > > > > > the state being altered. Some validation code talks to
> the
> > > >>>> MetaStore
> > > >>>> > > > > > directly, outside of the data produced by the Resolver.
> > > >>>> > > > > >
> > > >>>> > > > > > I do not think Polaris offers any explicit mechanisms
> > (ATM)
> > > >>>> to ensure
> > > >>>> > > > > > consistency between these reads and subsequent writes.
> > > >>>> > > > > >
> > > >>>> > > > > > As far as JDBC goes, running an overarching Transaction
> > > >>>> across all
> > > >>>> > > > reads
> > > >>>> > > > > > and writes in the same request, at the SERIALIZABLE
> > > isolation
> > > >>>> level
> > > >>>> > > in
> > > >>>> > > > > the
> > > >>>> > > > > > backing RDBMS could solve the problem, I think.
> > > >>>> > > > > >
> > > >>>> > > > > > Indeed, such a transaction might be long to accommodate
> > > calls
> > > >>>> made by
> > > >>>> > > > > > Polaris to external storage, etc. However, is that a
> > > problem?
> > > >>>> I think
> > > >>>> > > > the
> > > >>>> > > > > > alternative is for the JDBC Persistence impl. to
> > "manually"
> > > >>>> track all
> > > >>>> > > > > reads
> > > >>>> > > > > > under the same request and redo them in the small
> > > transaction
> > > >>>> that
> > > >>>> > > > > persists
> > > >>>> > > > > > the writes. This will also add RDBMS overhead and
> require
> > > >>>> complex
> > > >>>> > > code
> > > >>>> > > > in
> > > >>>> > > > > > Polaris to handle the data properly.
> > > >>>> > > > > >
> > > >>>> > > > > > Tracking the request-wide transaction can be done only
> for
> > > >>>> JDBC
> > > >>>> > > without
> > > >>>> > > > > > leaking "transaction" concepts to the NoSQL
> Persistence, I
> > > >>>> think.
> > > >>>> > > NoSQL
> > > >>>> > > > > > will use other mechanisms to ensure read/write
> > consistency.
> > > >>>> > > > > >
> > > >>>> > > > > > Connecting to Robert's email (a parallel branch in this
> > > >>>> discussion),
> > > >>>> > > > I'd
> > > >>>> > > > > > like to propose this approach:
> > > >>>> > > > > >
> > > >>>> > > > > > * Each request establishes a "Data Context"
> > > >>>> > > > > > - In JDBC the Data Context corresponds to a JDBC
> > > Connection
> > > >>>> + Tx
> > > >>>> > > > > > - In NoSQL the Data Context tracks one or more
> reference
> > > >>>> hashes
> > > >>>> > > > > > * All Persistence access in the same request goes
> through
> > > the
> > > >>>> same
> > > >>>> > > Data
> > > >>>> > > > > > Context
> > > >>>> > > > > > * All Persistence changes are committed once at the end
> of
> > > the
> > > >>>> > > request
> > > >>>> > > > > > - Not all changes have to be globally atomic. For
> > example,
> > > >>>> changes
> > > >>>> > > in
> > > >>>> > > > > > Catalogs A and B do not have to be atomic with respect
> to
> > > >>>> each other.
> > > >>>> > > > We
> > > >>>> > > > > > can go deeper into this later. This is relevant to
> NoSQL.
> > > >>>> > > > > > - Transactional backends like JDBC can, of course,
> > choose
> > > >>>> to make
> > > >>>> > > all
> > > >>>> > > > > > changes globally atomic.
> > > >>>> > > > > > * A commit can fail in two main ways:
> > > >>>> > > > > > - A retriable failure like an optimistic lock error or
> > Tx
> > > >>>> > > > > serializability
> > > >>>> > > > > > error
> > > >>>> > > > > > - A non-triable logical error (e.g. entity not found)
> > > >>>> > > > > > * On a retriable error the whole request is re-attempted
> > (as
> > > >>>> if
> > > >>>> > > > > resubmitted
> > > >>>> > > > > > by a client) a few times (configurable timeout).
> > > >>>> > > > > >
> > > >>>> > > > > > WDYT?
> > > >>>> > > > > >
> > > >>>> > > > > > Cheers,
> > > >>>> > > > > > Dmitri.
> > > >>>> > > > > >
> > > >>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré <
> > > >>>> > > jb@nanthrax.net
> > > >>>> > > > >
> > > >>>> > > > > > wrote:
> > > >>>> > > > > >
> > > >>>> > > > > > > Hi all
> > > >>>> > > > > > >
> > > >>>> > > > > > > I think Robert has a good point.
> > > >>>> > > > > > >
> > > >>>> > > > > > > If the atomicity guarantee lives only in
> > BasePersistence,
> > > >>>> then the
> > > >>>> > > > > > > manager contract can't tell a caller whether the state
> > it
> > > >>>> read for
> > > >>>> > > > > > > validation/authorization/credential-vending is the
> same
> > > >>>> state that
> > > >>>> > > > > > > eventually commits. That's the actual bug class behind
> > the
> > > >>>> JDBC
> > > >>>> > > > > > > symptoms (not any single operation being non-atomic,
> but
> > > the
> > > >>>> > > contract
> > > >>>> > > > > > > being silent about it. Every operation-specific fix
> > (like
> > > >>>> #4939,
> > > >>>> > > > #5035
> > > >>>> > > > > > > or #5095) re-answers this question locally and it
> keeps
> > > >>>> recurring.
> > > >>>> > > So
> > > >>>> > > > > > > the deliverable should start with a written
> consistency
> > > >>>> contract at
> > > >>>> > > > > > > the manager level.
> > > >>>> > > > > > >
> > > >>>> > > > > > > I'm not sure migrating everyone to
> > > >>>> > > TransactionalMetaStoreManagerImpl
> > > >>>> > > > > > > is a good idea. It would tie the logical change set
> to a
> > > DB
> > > >>>> > > > > > > transaction spanning the REST request. It means:
> > > >>>> > > > > > > - it holds a durable transaction open across slow
> > external
> > > >>>> work
> > > >>>> > > > > > > (credential vending, OPA, Ranger, ...)
> > > >>>> > > > > > > - it doesn't map to NoSQL
> > > >>>> > > > > > > - It wraps single-row updates in
> runWiithinTransaction,
> > > >>>> which is a
> > > >>>> > > > > > overhead
> > > >>>> > > > > > >
> > > >>>> > > > > > > So, I think the transactional manager isn't a portable
> > > >>>> target. It's
> > > >>>> > > > > > > "only" one backend's strategy.
> > > >>>> > > > > > >
> > > >>>> > > > > > > I think Privthi's approach is right. A
> backend-agnostic
> > > >>>> change-set
> > > >>>> > > > > > > primitive with a documented fallback is the correct
> > shape.
> > > >>>> There is
> > > >>>> > > > > > > one caveat: without carrying the original entitiy (not
> > > just
> > > >>>> the new
> > > >>>> > > > > > > one) the change set can't express optimistic
> > concurrency.
> > > >>>> > > > > > >
> > > >>>> > > > > > > I propose the following multi-steps approach:
> > > >>>> > > > > > > 1. We write the manager-level consistency contract as
> > > >>>> Robert asked.
> > > >>>> > > > It
> > > >>>> > > > > > > should include the explicit statement that a logical
> > > change
> > > >>>> set is
> > > >>>> > > > not
> > > >>>> > > > > > > a request-scope DB transaction.
> > > >>>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency
> > > pattern)
> > > >>>> > > baseline
> > > >>>> > > > > > > first-class in EntityMutation before merging the SPI
> > > >>>> > > > > > > 3. Then we refactor
> > createCatalog/dropEntity/renameEntity.
> > > >>>> > > > > > >
> > > >>>> > > > > > > Thoughts?
> > > >>>> > > > > > >
> > > >>>> > > > > > > Regards
> > > >>>> > > > > > > JB
> > > >>>> > > > > > >
> > > >>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <
> > > >>>> snazy@snazy.de>
> > > >>>> > > wrote:
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > Hi all,
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > Yufei's clarification about where the atomicity
> > > guarantee
> > > >>>> is
> > > >>>> > > > defined
> > > >>>> > > > > > > seems
> > > >>>> > > > > > > > important.
> > > >>>> > > > > > > > If it is a property of the lower-level
> BasePersistence
> > > >>>> contract
> > > >>>> > > > > rather
> > > >>>> > > > > > > than
> > > >>>> > > > > > > > the general PolarisMetaStoreManager contract, the
> > > general
> > > >>>> > > contract
> > > >>>> > > > > does
> > > >>>> > > > > > > not
> > > >>>> > > > > > > > tell callers whether the state used for validation,
> > > >>>> > > authorization,
> > > >>>> > > > or
> > > >>>> > > > > > > > credential vending is consistent with the change
> that
> > > >>>> eventually
> > > >>>> > > > > > commits.
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > The current PRs suggest that operation-specific
> > > >>>> multi-object
> > > >>>> > > > methods
> > > >>>> > > > > > can
> > > >>>> > > > > > > > fix individual cases, while leaving the same
> contract
> > > >>>> question to
> > > >>>> > > > > recur
> > > >>>> > > > > > > for
> > > >>>> > > > > > > > each new case.
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > I would also avoid defining a logical change set as
> a
> > > >>>> database
> > > >>>> > > > > > > transaction
> > > >>>> > > > > > > > around an entire REST request.
> > > >>>> > > > > > > > That would tie the contract to the backend and
> > database,
> > > >>>> and
> > > >>>> > > could
> > > >>>> > > > > keep
> > > >>>> > > > > > > the
> > > >>>> > > > > > > > durable attempt open across slow external work.
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > So is the choice really between the two current
> > manager
> > > >>>> > > > > > implementations,
> > > >>>> > > > > > > or
> > > >>>> > > > > > > > do we first need to revisit the boundary and
> > guarantees
> > > >>>> exposed
> > > >>>> > > to
> > > >>>> > > > > > their
> > > >>>> > > > > > > > callers?
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > Cheers,
> > > >>>> > > > > > > > Robert
> > > >>>> > > > > > > >
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri
> Bourlatchkov <
> > > >>>> > > > > dimas@apache.org
> > > >>>> > > > > > >
> > > >>>> > > > > > > > wrote:
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > > Hi Yufei,
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > Thanks for the link. I stand corrected. The "one
> > > atomic
> > > >>>> change
> > > >>>> > > > per
> > > >>>> > > > > > > method"
> > > >>>> > > > > > > > > contract as defined in javadoc does apply only to
> > > >>>> > > BasePersistence
> > > >>>> > > > > > > > > and AtomicOperationMetaStoreManager (which
> delegates
> > > to
> > > >>>> > > > > > > BasePersistence).
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > Note that all non-test Persistence implementations
> > in
> > > >>>> the
> > > >>>> > > Polaris
> > > >>>> > > > > > > codebase
> > > >>>> > > > > > > > > extend those classes (which is probably why I was
> > > >>>> confused
> > > >>>> > > about
> > > >>>> > > > > > > atomicity
> > > >>>> > > > > > > > > expectations).
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > However, this creates a gap in the Persistence SPI
> > > >>>> > > specification.
> > > >>>> > > > > If
> > > >>>> > > > > > > > > other PolarisMetaStoreManager implementations do
> not
> > > >>>> have to
> > > >>>> > > > comply
> > > >>>> > > > > > > with
> > > >>>> > > > > > > > > this principle, it will create a conceptual
> > difficulty
> > > >>>> at call
> > > >>>> > > > > sites.
> > > >>>> > > > > > > How
> > > >>>> > > > > > > > > can PolarisMetaStoreManager callers reason about
> > > >>>> consistency
> > > >>>> > > and
> > > >>>> > > > > > > durability
> > > >>>> > > > > > > > > behaviours in general?
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > I believe we need to address that as part of this
> > > >>>> discussion.
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > Cheers,
> > > >>>> > > > > > > > > Dmitri.
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <
> > > >>>> flyrain000@gmail.com
> > > >>>> > > >
> > > >>>> > > > > > wrote:
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with
> the
> > > >>>> idea that
> > > >>>> > > one
> > > >>>> > > > > > > method
> > > >>>> > > > > > > > > call
> > > >>>> > > > > > > > > > means one atomic change.
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > > If the "MetaStore SPI" refers to the interface
> > > >>>> > > > > > > PolarisMetaStoreManager, I
> > > >>>> > > > > > > > > > don't think we've ever state each method to be
> > > >>>> atomic. We did
> > > >>>> > > > > > clarify
> > > >>>> > > > > > > > > > atomicity[1] in the interface BasePersistence
> > > though.
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > > 1.
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > >
> > > >>>> > > > > > >
> > > >>>> > > > > >
> > > >>>> > > > >
> > > >>>> > > >
> > > >>>> > >
> > > >>>>
> > >
> >
> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > > Yufei
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri
> > > Bourlatchkov <
> > > >>>> > > > > > > dimas@apache.org>
> > > >>>> > > > > > > > > > wrote:
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > > > Hi Yufei,
> > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > > > I agree that JDBC transactions must be handled
> > > more
> > > >>>> > > > explicitly.
> > > >>>> > > > > > > > > However,
> > > >>>> > > > > > > > > > > I'm not sure that simply moving to
> > > >>>> > > > > > > TransactionalMetaStoreManagerImpl is
> > > >>>> > > > > > > > > > > sufficient.
> > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with
> the
> > > >>>> idea that
> > > >>>> > > one
> > > >>>> > > > > > > method
> > > >>>> > > > > > > > > call
> > > >>>> > > > > > > > > > > means one atomic change [1]. The
> "transactional"
> > > >>>> MetaStore
> > > >>>> > > > > impl.
> > > >>>> > > > > > is
> > > >>>> > > > > > > > > but a
> > > >>>> > > > > > > > > > > sub-case of that. It cannot alter the
> high-level
> > > >>>> contract.
> > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > > > We could add SPI methods having multiple
> object
> > > >>>> parameters
> > > >>>> > > to
> > > >>>> > > > > > > represent
> > > >>>> > > > > > > > > > > grouped changes, but I am not sure it will be
> a
> > > >>>> sound
> > > >>>> > > design.
> > > >>>> > > > > > This
> > > >>>> > > > > > > will
> > > >>>> > > > > > > > > > > bloat the interface surfaces and require extra
> > > >>>> impl. effort
> > > >>>> > > > for
> > > >>>> > > > > > > each
> > > >>>> > > > > > > > > > > backend type. More importantly, adding
> multi-arg
> > > >>>> change
> > > >>>> > > > methods
> > > >>>> > > > > > > still
> > > >>>> > > > > > > > > > won't
> > > >>>> > > > > > > > > > > address the problem of reads being consistent
> > with
> > > >>>> writes,
> > > >>>> > > > > > because
> > > >>>> > > > > > > each
> > > >>>> > > > > > > > > > > method call will still be independent
> regarding
> > > the
> > > >>>> data
> > > >>>> > > > stored
> > > >>>> > > > > > in
> > > >>>> > > > > > > the
> > > >>>> > > > > > > > > > > database.
> > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > > > I tend to think we need to introduce a "change
> > > set"
> > > >>>> or
> > > >>>> > > > "atomic
> > > >>>> > > > > > > batch"
> > > >>>> > > > > > > > > > > concept to core Persistence and associate each
> > > REST
> > > >>>> API
> > > >>>> > > > request
> > > >>>> > > > > > > with
> > > >>>> > > > > > > > > one
> > > >>>> > > > > > > > > > > such change set, which will be committed (or
> > > rolled
> > > >>>> back)
> > > >>>> > > at
> > > >>>> > > > > the
> > > >>>> > > > > > > end of
> > > >>>> > > > > > > > > > the
> > > >>>> > > > > > > > > > > request. I believe Ayush mentioned a similar
> > > >>>> concept in PR
> > > >>>> > > > 4939
> > > >>>> > > > > > > [2]. In
> > > >>>> > > > > > > > > > > JDBC each change set will naturally be
> > associated
> > > >>>> with an
> > > >>>> > > > RDBMS
> > > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each atomic
> > > >>>> change set
> > > >>>> > > > will
> > > >>>> > > > > be
> > > >>>> > > > > > > > > > > associated with one CAS operation on the
> > > underlying
> > > >>>> > > database.
> > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > > > [1]
> > > >>>> > > > > > >
> > > >>>> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
> > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > > > [2]
> > > >>>> > > > > > >
> > > >>>>
> https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > > > Cheers,
> > > >>>> > > > > > > > > > > Dmitri.
> > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
> > > >>>> > > > flyrain000@gmail.com
> > > >>>> > > > > >
> > > >>>> > > > > > > wrote:
> > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > > > > Thanks for raising this, Dmitri.
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > These are valid concerns, and they were
> > already
> > > >>>> > > recognized
> > > >>>> > > > > when
> > > >>>> > > > > > > we
> > > >>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At
> > that
> > > >>>> time, we
> > > >>>> > > > > chose
> > > >>>> > > > > > > to use
> > > >>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC
> > due
> > > >>>> to the
> > > >>>> > > > > > > simplicity. I
> > > >>>> > > > > > > > > > > think
> > > >>>> > > > > > > > > > > > most of the issues mentioned here can
> already
> > be
> > > >>>> > > addressed
> > > >>>> > > > by
> > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl.
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > For example, rename is already wrapped in a
> > > >>>> transaction
> > > >>>> > > in
> > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1].
> > > Similarly,
> > > >>>> catalog
> > > >>>> > > > > > > creation,
> > > >>>> > > > > > > > > > which
> > > >>>> > > > > > > > > > > > involves reading and creating multiple
> > objects,
> > > >>>> is also
> > > >>>> > > > > > executed
> > > >>>> > > > > > > > > > within a
> > > >>>> > > > > > > > > > > > transaction [2].
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > I see two possible directions:
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > 1.
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager
> > > >>>> together with
> > > >>>> > > the
> > > >>>> > > > > > > > > persistence
> > > >>>> > > > > > > > > > > > backends (such as JDBC) to provide the
> > > required
> > > >>>> > > > > consistency
> > > >>>> > > > > > > > > > guarantees
> > > >>>> > > > > > > > > > > > for
> > > >>>> > > > > > > > > > > > specific operations, similar to what
> > > >>>> > > > > > > > > > TransactionalMetaStoreManagerImpl
> > > >>>> > > > > > > > > > > > does.
> > > >>>> > > > > > > > > > > > 2.
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > Migrate the persistence backends (such as
> > > >>>> JDBC) to use
> > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl
> directly.
> > > We
> > > >>>> may
> > > >>>> > > have
> > > >>>> > > > to
> > > >>>> > > > > > > deal
> > > >>>> > > > > > > > > with
> > > >>>> > > > > > > > > > > > transactional semantic mismatches across
> > > >>>> different
> > > >>>> > > > > > persistence
> > > >>>> > > > > > > > > > > backends.
> > > >>>> > > > > > > > > > > > For example, we would likely avoid using
> > > JDBC's
> > > >>>> > > > > > > > > > `runWithinTransaction`
> > > >>>> > > > > > > > > > > > for
> > > >>>> > > > > > > > > > > > single row updates, which adds additional
> > > >>>> overhead and
> > > >>>> > > > > > > complexity
> > > >>>> > > > > > > > > > > > without
> > > >>>> > > > > > > > > > > > benefits.
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > References:
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > 1.
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > >
> > > >>>> > > > > > >
> > > >>>> > > > > >
> > > >>>> > > > >
> > > >>>> > > >
> > > >>>> > >
> > > >>>>
> > >
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > > >>>> > > > > > > > > > > > 2.
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > >
> > > >>>> > > > > > >
> > > >>>> > > > > >
> > > >>>> > > > >
> > > >>>> > > >
> > > >>>> > >
> > > >>>>
> > >
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > Yufei
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri
> > > >>>> Bourlatchkov <
> > > >>>> > > > > > > > > dimas@apache.org>
> > > >>>> > > > > > > > > > > > wrote:
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > Hi all,
> > > >>>> > > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a
> > > couple
> > > >>>> of
> > > >>>> > > > > > interesting
> > > >>>> > > > > > > PRs:
> > > >>>> > > > > > > > > > > > > [4939], [5035].
> > > >>>> > > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > It looks like people are starting to
> > encounter
> > > >>>> > > > consistency
> > > >>>> > > > > > > issues
> > > >>>> > > > > > > > > in
> > > >>>> > > > > > > > > > > > > JDBC persistence.
> > > >>>> > > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > The PRs provide valuable insight into the
> > > >>>> underlying
> > > >>>> > > > > issues.
> > > >>>> > > > > > > They
> > > >>>> > > > > > > > > > offer
> > > >>>> > > > > > > > > > > > > incremental fixes that can work. However,
> I
> > > >>>> believe it
> > > >>>> > > is
> > > >>>> > > > > > time
> > > >>>> > > > > > > for
> > > >>>> > > > > > > > > > the
> > > >>>> > > > > > > > > > > > > Polaris community to review and improve
> this
> > > >>>> area of
> > > >>>> > > the
> > > >>>> > > > > > > codebase
> > > >>>> > > > > > > > > > > > > holistically.
> > > >>>> > > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > By this, I mean finding a solution that
> can
> > be
> > > >>>> applied
> > > >>>> > > to
> > > >>>> > > > > all
> > > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC,
> > NoSQL)
> > > >>>> and
> > > >>>> > > > addresses
> > > >>>> > > > > > > these
> > > >>>> > > > > > > > > > > > > aspects:
> > > >>>> > > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent
> > changes
> > > >>>> where
> > > >>>> > > the
> > > >>>> > > > > > > service
> > > >>>> > > > > > > > > > reads
> > > >>>> > > > > > > > > > > > > and validates current catalog state,
> then
> > > >>>> commits a
> > > >>>> > > > > change
> > > >>>> > > > > > > (e.g.
> > > >>>> > > > > > > > > > > > > name clashes during renames).
> > > >>>> > > > > > > > > > > > > * Supporting consistent but independent
> > > changes
> > > >>>> to RBAC
> > > >>>> > > > > > grants
> > > >>>> > > > > > > and
> > > >>>> > > > > > > > > > > > > MetaStore entities. This independence is
> > > >>>> needed to
> > > >>>> > > > > support
> > > >>>> > > > > > > > > > > > > external authorizers like OPA and
> Ranger.
> > > >>>> > > > > > > > > > > > > * Supporting atomic changes across
> multiple
> > > >>>> similar
> > > >>>> > > > > entities.
> > > >>>> > > > > > > > > > > > > * Supporting authorization-based filtering
> > of
> > > >>>> list
> > > >>>> > > > > operations
> > > >>>> > > > > > > (cf.
> > > >>>> > > > > > > > > > > > > [4831]).
> > > >>>> > > > > > > > > > > > > * Supporting credential-vending decisions
> > that
> > > >>>> are
> > > >>>> > > rooted
> > > >>>> > > > > in
> > > >>>> > > > > > > the
> > > >>>> > > > > > > > > > > > > exact state of the catalog.
> > > >>>> > > > > > > > > > > > > * Supporting server-side retries for
> > transient
> > > >>>> > > > persistence
> > > >>>> > > > > > > failures
> > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability
> failures).
> > > >>>> > > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > Please share your comments and ideas.
> > > >>>> > > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > [4831]
> > > >>>> https://github.com/apache/polaris/pull/4831
> > > >>>> > > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > [4939]
> > > >>>> https://github.com/apache/polaris/pull/4939
> > > >>>> > > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > [5035]
> > > >>>> https://github.com/apache/polaris/pull/5035
> > > >>>> > > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > Thanks,
> > > >>>> > > > > > > > > > > > > Dmitri
> > > >>>> > > > > > > > > > > > >
> > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > >
> > > >>>> > > > > > >
> > > >>>> > > > > >
> > > >>>> > > > >
> > > >>>> > > >
> > > >>>> > >
> > > >>>>
> > > >>>
> > >
> >
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Ankur Kotwal <ko...@gmail.com>.
Hi EJ, all,
I tried the PoC from the angle of the catalog-branches thread, since
multi-table promotion depends on exactly this guarantee.
I took the multi-table promotion test from #5740 (branch three tables, write
to the branch, then move every table's main in one transactions/commit that
asserts main hasn't moved) and ran it on your branch with
PrimitiveServiceTestFactory (H2), plus two race tests:
- two promotions of the same tables from the same base, started together,
10 rounds: exactly one won every round, and the tables never showed a mix
of the two;
- a promotion racing a single-table commit to main of one of the tables,
both asserting main is still at the base, 30 rounds with a random delay so
both orders occur (the promotion won 11, the writer 19): exactly one
succeeded each time, and a failed promotion left every table unchanged.
All four pass on your branch, and the same four pass on main with the
in-memory metastore, so for this path the new layer keeps today's
guarantees. As your README says, H2 doesn't tell us much about PostgreSQL
or CockroachDB under serializable isolation; I didn't have a database to
run them against here. If it's useful, I can send the test as a PR to your
branch so it runs in your PostgreSQL and CockroachDB jobs.
Three things I noticed while reading the code:
1. updateEntitiesPropertiesIfNotChanged now validates the whole batch before
writing, and rejects a batch that contains the same entity twice with
TARGET_ENTITY_CONCURRENTLY_MODIFIED. Rejecting it makes sense, but in
commitTransaction that status ends up as a CommitFailedException (409),
which Iceberg clients treat as a conflict and retry, and the retry can
never succeed. A distinct status, or rejecting duplicate identifiers
with a 400 before the commit, might be clearer.
2. Running the location-overlap check over the whole batch inside the same
attempt closes the race between two requests that each see no overlap.
I like that this lives once in the Manager rather than in each backend.
3. For catalog branches (the NoSQL named-pointer option in the other
thread), the Primitives contract is enough as it stands: a branch head
is one more key, and publishing is one attempt that reads the head and
the target and commits the new pointer. Is the plan for the NoSQL
persistence to stay on its own Persistence layer, or could it eventually
sit on Primitives too? That would decide where the branch registry
should live.
Thanks for putting this together,
Ankur
On Wed, Oct 07, 2026 08:57 PM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi guys,
>
> Thanks a lot EJ for the updated PoC and testing it across multiple
> backends.
>
> At a high level, I really like the direction and separation of concerns
> here:
> - Having a business logic centralized in the shared Manager rathe than
> re-implemented across backends avoids logic drift and fixes the maintenance
> burden we have today. I remember it was my call when we had Community
> Meetings were we discussed OpenJPA -> JDBC and later NoSQL.
> - Keeping the storage adapter contract focused on basic storage operations
> without "coupling" in table/catalog concepts is clean and portable (I have
> the feeling it was discussed while back as well :)).
> - Ensuring STS and object storage calls stay decoupled from the database
> transaction boundary is essential.
>
> I will take a new look on your branch, we probably need a couple of weeks
> or so (several of us are travelling to CoC).
>
> Personally, I'm interested in closely examining the failure handling around
> uncertain commits.
>
> Thanks!
>
> Regards
> JB
>
> On Thu, Oct 8, 2026 at 5:33 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi EJ,
> >
> > Thanks for sharing the PoC!
> >
> > I'll try to explore that code, but cannot promise a quick turnaround :)
> > Reviewing this may take me a couple of weeks (due to other priorities).
> >
> > Cheers,
> > Dmitri.
> >
> > On Mon, Oct 5, 2026 at 4:47 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> > > One detail I omitted: the PoC also includes FoundationDB and Spanner
> > > adapters alongside JDBC. They reuse the same business-aware Manager.
> The
> > > adapters implement generic storage operations without table, catalog or
> > > grant rules.
> > >
> > > This exercises both goals: addressing consistency problems such as
> > > concurrent creation at overlapping table locations, and reusing that
> > logic
> > > across databases. The concurrency tests have run on FoundationDB,
> > > PostgreSQL and CockroachDB. Spanner coverage currently uses the
> emulator,
> > > with three concurrency tests skipped.
> > >
> > > These are targeted feasibility checks. Production validation and
> > > integration with Polaris’s existing NoSQL backend remain follow-up
> work.
> > >
> > > -ej
> > >
> > > On Mon, Oct 5, 2026 at 1:36 PM EJ Wang <ej.wang.dev.interest@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Dmitri, all,
> > > >
> > > > Two requests can both check for overlapping table locations, see
> none,
> > > and
> > > > then create conflicting tables. We need the checks and the resulting
> > > writes
> > > > to be protected together.
> > > >
> > > > My proposal is to implement these rules once in shared Polaris code
> > (the
> > > > Manager). This business-aware layer checks them in the transaction
> that
> > > > creates the table. Database adapters would implement a
> > business-agnostic
> > > > interface (Primitives) for generic reads, writes and commit. They
> would
> > > not
> > > > implement table, catalog or grant rules, so adding a database reuses
> > the
> > > > same Polaris behavior.
> > > >
> > > > This PoC assumes the database itself can protect the relevant reads
> and
> > > > writes in one serializable transaction. Calls to object storage or
> > > > credential services happen outside it.
> > > >
> > > > The Java PoC
> > > > <
> > >
> > https://github.com/flyingImer/polaris/tree/prototype/durable
> -java-primitives/persistence/primitives
> > > >
> > > > reuses the existing transactional Manager and tests selected races,
> > > > authorization retries and uncertain commits. It uses a temporary data
> > > > layout and reads more data than we'd want in production. Existing
> NoSQL
> > > > integration, production tuning and migration remain follow-up work.
> > > >
> > > > I'm not proposing an additional request-wide Persistence Session yet.
> > > > Existing business-aware request code still owns authorization,
> external
> > > > work and retries.
> > > >
> > > > The PoC allows an already-authorized metadata change to finish. A
> fresh
> > > > request retry checks authorization again. An uncertain commit stops
> > > > automatic retries and retains files that may already be referenced.
> > These
> > > > policies still need agreement.
> > > >
> > > > Would this separation be a useful starting point?
> > > >
> > > > -ej
> > > >
> > > >
> > > > On Tue, 22 Sep 2026 19:08:49 -0400, Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > wrote:
> > > >
> > > > Hi EJ, > 1. Which backend assumptions should we design for? [...] I
> > know
> > > I
> > > > used the term "transaction" a lot during the call, but the real
> > challenge
> > > > is about establishing an SPI with clear contracts that could be
> > > implemented
> > > > by JDBC and NoSQL persistence using the database features natural to
> > each
> > > > model. I am fairly confident that a JDBC impl. is possible with
> > > > Serializable transaction. We should be able to solve all the problems
> > > > raised in this thread. However, that will make in-memory caching
> > > impossible
> > > > because the database needs to see all the reads (and writes) in a Tx
> to
> > > be
> > > > able to ensure serializable guarantees. If a cache prevents a read
> from
> > > > hitting the database, that read (e.g. storage config) will not be a
> > > factor
> > > > in serializability. This connects to my earlier question about the
> > > > in-memory cache and JDBC (title: Usefulness of InMemoryEntityCache)
> Are
> > > > different approaches possible with JDBC? Probably. What is involved?
> > This
> > > > depends on the SPI contract, I think. > 2. How do we protect the data
> > > used
> > > > by validation? [...] The point mentioned by Robert previously IIRC,
> is
> > > that
> > > > we need to ground all runtime decisions to a particular state of the
> > > > catalog. If at commit time the state of the catalog in the database
> > > shifts
> > > > from what was used during validation, the request has to restart from
> > > > scratch. Current JDBC Persistence does this only for relationships
> > > between
> > > > entities and grants and between versions of the same entity.
> > > Relationships
> > > > between table locations and storage config are not actualized for the
> > > > purpose of state consistency tracking. From my POV we need to begin
> by
> > > > formulating this Persistence state in the Persistence SPI terms.
> Then,
> > > > think how we can implement that for JDBC. NoSQL Persistence already
> > has a
> > > > per-catalog state (not granular), so wiring the new SPI to NoSQL
> should
> > > not
> > > > be too hard, I hope. The state in the SPI does not have to be global
> to
> > > the
> > > > catalog (as in NoSQL), but it needs to be able to express the idea of
> > > which
> > > > elements were engaged during request processing so serve as input to
> > > > Persistence impl. for cosistency guarantees. If Persistence caches
> data
> > > in
> > > > memory (for example), it will have to do so in a way that does not
> > > violate
> > > > consistency expectations. From my POV we could begin by defining a
> > > > "Persistence Session" concept. Each request is 1:1 with a session.
> All
> > > > reads and writes go through the session. The next step would be the
> > JDBC
> > > > impl., which probably needs some brainstorming... but I wonder if we
> > > could
> > > > reach a consensus at a high level first. WDYT? Thanks, Dmitri. On
> Tue,
> > > Sep
> > > > 22, 2026 at 5:55 PM EJ Wang <ej...@gmail.com> wrote:
> >
> > > > Thanks for the recap, Dmitri. I'd like to follow up on the three
> points
> > > I >
> > > > raised during the sync. We didn't have much time to discuss them, so
> > it >
> > > > would help to confirm these before choosing an SPI design. > > 1.
> Which
> > > > backend assumptions should we design for? > > Within one Polaris
> server
> > > > instance, do we assume entities, grants and > secrets all use the
> same
> > > > database backend, or must we support mixing > backends across them? >
> > > Must
> > > > every supported persistence implementation provide transactional >
> > > > guarantees? Does the database itself need to provide transactions, or
> > > can >
> > > > Polaris build that capability on top of simpler database operations?
> >
> > >
> > > > I'm happy to work with a homogeneous assumption if that's the
> intended
> > >
> > > > scope. Existing NoSQL illustrates the second question: it can
> publish >
> > > > changes to multiple objects under one reference using a single
> > > conditional
> > > > > pointer update. > > 2. How do we protect the data used by
> validation?
> > > > >
> > > > Your storage-configuration and location-overlap examples make the
> > > problem >
> > > > concrete. Consider these races: > > A table creation passes
> validation
> > > > against the catalog's allowed locations, > then another request
> changes
> > > > that configuration before the table is > committed. > Two
> > table-creation
> > > > requests each check for overlapping locations, both see > no overlap,
> > and
> > > > then create tables at conflicting locations. > > Checking only the
> > record
> > > > being written cannot cover these dependencies. The > first also
> depends
> > > on
> > > > the catalog configuration. The second depends on the > absence of
> > another
> > > > overlapping table. > > Could the persistence layer protect these
> reads
> > > > together with the writes, > using backend-internal versions/tokens or
> > > > database isolation? Shared code > would still need to identify the
> > > required
> > > > reads and checks. Could we keep > the mechanism for protecting them
> > > inside
> > > > each backend? > > 3. Business rules are duplicated across backend
> > > > implementations > > Today, the TreeMap > < >
> > > >
> > >
> > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0
> 663222ce2ee23621917/polaris-core/src/main/java/org/apache/
> polaris/core/persistence/transactional/TreeMapTransactionalP
> ersistenceImpl.java#L446-L481
> > > > > > > and JDBC > < >
> > > >
> > >
> > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0
> 663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/
> polaris/persistence/relational/jdbc/JdbcBasePersis
> tenceImpl.java#L1002-L1046
> > > > > > > implementations each load a secret, check its principal, apply
> > the
> > > >
> > > > rotation/reset steps and write it back. > > Changing these business
> > rules
> > > > therefore requires keeping multiple > implementations in sync. A fix
> > > > applied to one backend can be missed in > another, causing the same
> > > Polaris
> > > > operation to behave differently depending > on the database. > > My
> > > > proposal is to implement each business workflow once, against a
> common
> > >
> > > > set of storage operations with defined consistency and failure
> > > semantics. >
> > > > The shared workflow should not need to know whether those operations
> > are
> > > >
> > > > implemented through JDBC transactions, NoSQL reference CAS or
> another >
> > > > mechanism. Each backend would implement the same storage contract,
> > > keeping
> > > > > its execution details internal. A new database backend could then
> > reuse
> > > > the > existing business workflows. > > Do these seem like reasonable
> > > > starting points? We can then evaluate the SPI > against the original
> > > > failure cases on both JDBC and existing NoSQL. > > -ej > > On Fri,
> Sep
> > > 18,
> > > > 2026 at 12:10 PM Dmitri Bourlatchkov <di...@apache.org> > wrote: >
> > >
> > > > Side note: I just came across [5541], which shows a case where a
> failed
> > > > >
> > > > request leaves persisted side effects. > > > > This can be seen as a
> > > coding
> > > > mistake, of course. However, this is about > > executing one request
> > as a
> > > > single, atomic unit. This is something that > > probably needs
> > > foundational
> > > > support and coding patterns in Polaris code. > > > > Cheers, > >
> > Dmitri.
> > > >
> > > > > [5541] https://github.com/apache/polaris/pull/5541 > > > > On Fri,
> > Sep
> > > > 18, 2026 at 3:02 PM Dmitri Bourlatchkov <di...@apache.org> > >
> wrote:
> > >
> > > >
> > > > > > > Hi All, > > > > > > Here's a recap of the Community Sync
> > discussion
> > > > as I remember it. I'm > > sure > > > my recollection is not complete,
> > so
> > > > please add your thoughts to this > > thread. > > > > > > * Polaris
> > > > traditionally relies on entity version number to ensure > > >
> > consistency
> > > > of changes. > > > > > > This works well for internal RBAC grants,
> which
> > > > also have version > numbers > > > cross-referenced at the Persistence
> > > > layer. > > > > > > I do not think this works in more general cases
> like
> > > > validating > locations > > > wrt catalog-level storage
> configuration. >
> > > > >
> > > > > > > Another difficult case involves location overlaps between
> > > concurrent
> > > > > > table > > > creations. > > > > > > * We talked about how
> > Persistence
> > > > calls related to RDBMS transactions > (in > > > the JDBC persistence
> > > case).
> > > > > > > > > > There were some concerns raised about whether we need to
> > > bind >
> > > > Persistence > > > API to transactions explicitly. > > > > > > I think
> > it
> > > is
> > > > a valid concern. At the same time, the Persistence API > > > should
> be
> > > > clear about consistency guarantees across all backend > > >
> > > > implementations. I think this needs more attention now that the
> matter
> > >
> > > of
> > > > > > > milti-entioty changes became prominent in [5035]. > > > > > > I
> > > > personally believe that RDBMS-based Persistence in Polaris has to
> use >
> > > > >
> > > > Serializable transaction isolation. Consequently, the core code
> needs >
> > > to
> > > > > > > provide mechanisms to allow the Persistence implementation to
> > > > properly > > > connect reads and writes from API requests to JDBC
> > > > transactions... > > whether > > > we use "transation" as an explicit
> > > terms
> > > > in the SPI or not. > > > > > > Note that currently, the same API
> > request
> > > > performs reads and writes in > > > _separate_ RDBMS transactions
> (even
> > > > separate connections). > > > > > > * We did not discuss Persistence
> SPI
> > > > implications in the call, but > > > currently each Persistence call
> is
> > > > assumed to be one distinct and > atomic > > > change (please correct
> me
> > > if
> > > > I'm wrong). > > > > > > Consequently, supporting new use cases
> involves
> > > > introducing new SPI > > > methods with many parameters. This can be
> > seen
> > > in
> > > > PR [5035]. This tends > > to > > > overcomplicate the SPI. > > > > >
> >
> > > > Ideally, I think the SPI should be simplified to deal with
> multi-entity
> > > > >
> > > > > and single-entity changes in a coherent manner to avoid any
> ambiguity
> > > > >
> > > > about > > > which method the caller should use. > > > > > > [5035]
> > > > https://github.com/apache/polaris/pull/5035 > > > > > > Cheers, > >
> >
> > > > Dmitri. > > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri
> > Bourlatchkov
> > > <
> > > > dimas@apache.org > > > > > wrote: > > > > > >> Heads up: This
> > discussion
> > > > is on the Community Sync call agenda for > Sept > > >> 17. > > >> > >
> > >>
> > > > Interested parties, please plan to attend, if possible :) > > >> > >
> >>
> > > > Cheers, > > >> Dmitri. > > >> > > >> On Wed, Sep 9, 2026 at 1:48 PM
> > > Dmitri
> > > > Bourlatchkov <di...@apache.org> > > >> wrote: > > >> > > >>> Hi JB,
> >
> > >
> > > > >>> > > >>> I've added an agenda item to the next community sync call
> > for
> > > > this > > (Sept > > >>> 17, 2026). > > >>> > > >>> Re: key points: did
> > you
> > > > mean a real GH discussion or dev email (this > > >>> thread)? Just
> > double
> > > > checking :) I'm fine with either approach. > > >>> > > >>> Cheers, >
> >
> > > >>>
> > > > Dmitri. > > >>> > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste
> > > > Onofré < > jb@nanthrax.net> > > >>> wrote: > > >>> > > >>>> Hi
> > Dmitri, >
> > > > > >>>> > > >>>> I agree on the need to align on these points, though
> > I'm
> > > > not > entirely > > >>>> sure a dedicated meeting is necessary. Let's
> > > start
> > > > by using some > time > > >>>> during the next community meeting to
> > > discuss
> > > > it. > > >>>> > > >>>> In the meantime, what if we outline the key
> > points
> > > in
> > > > a GitHub > > >>>> Discussion first? We can then use that as a
> reference
> > > > during the > > >>>> meeting. > > >>>> > > >>>> Regards, > > >>>> JB
> > >
> > > > >>>> > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov < >
> > > > dimas@apache.org > > > > > >>>> wrote: > > >>>> > > > >>>> > Hi All,
> > > >
> > > > >>>> > > > >>>> > My impression from this thread is that it might be
> > time
> > > > for a call > > to > > >>>> > discuss all the related issues and try
> to
> > > > arrive at a shared > > >>>> implementation > > >>>> > plan. > > >>>>
> >
> > >
> > > >
> > > > >>>> > I know meetings are not ideal, but in this case it might be >
> >
> > > >>>>
> > > > beneficial as a > > >>>> > means for achieving a common understanding
> > of
> > > > the set of problems > > and > > >>>> > priorities related to this
> > > thread. A
> > > > dedicated metrics meeting > > worked > > >>>> well > > >>>> > from my
> > > POV.
> > > > > > >>>> > > > >>>> > Allocating a time slot in the next community
> > > meeting
> > > > might be an > > >>>> option, > > >>>> > although I think we might
> need
> > > the
> > > > full hour in this case. > > >>>> > > > >>>> > WDYT? > > >>>> > > >
> > >>>> >
> > > > Thanks, > > >>>> > Dmitri. > > >>>> > > > >>>> > On Mon, Aug 17, 2026
> > at
> > > > 6:39 PM Prithvi S < > > >>>> prithvisivasankar@gmail.com> > > >>>> >
> > > > wrote: > > >>>> > > > >>>> > > Hi all, > > >>>> > > > > >>>> > > I
> > > rewrote
> > > > https://github.com/apache/polaris/pull/5035, > following > > >>>>
> the
> > >
> > > >
> > > > >>>> > > discussion to focus only on the SPI foundation. > > >>>> > >
> > > >
> > > > >>>> > > What is in the PR now: > > >>>> > > > > >>>> > > - A written
> > > > manager-level consistency contract in > > >>>> > >
> > > > site/content/in-dev/unreleased/persistence-consistency.md. > > >>>>
> >
> > > -
> > > > CAS-aware EntityMutation / GrantMutation records. > > >>>> > > -
> > > > MetaStoreChangeSet and BasePersistence#commitChangeSet, > with > >
> >>>>
> > > > backends > > >>>> > > opting in via supportsAtomicMixedCommit(). > >
> > > >>>> >
> > > > > - commitChangeSet implementations for the in-memory TreeMap > >
> >>>>
> > > > backend and > > >>>> > > JDBC. > > >>>> > > -
> > > > AtomicOperationMetaStoreManager and > > >>>>
> > > > TransactionalMetaStoreManagerImpl > > >>>> > > grant/revoke paths
> now
> > > build
> > > > one change set per operation, > > >>>> falling back > > >>>> > > to
> > >
> > > > >>>> > > individual operations when the backend does not support
> mixed
> > >
> > > >
> > > > >>>> commits. > > >>>> > > - Tests for the change-set API and TreeMap
> > > > atomic commits. > > >>>> > > > > >>>> > > I intentionally left out:
> > >
> > > > >>>> > > > > >>>> > > - Entity deletes in MetaStoreChangeSet. > >
> >>>>
> > >
> > > >
> > > > - Refactoring createCatalog, dropEntity, and renameEntity to > > use
> >
> > >
> > > > >>>> change > > >>>> > > sets. > > >>>> > > - NoSQL support for
> > > > commitChangeSet. > > >>>> > > > > >>>> > > I am treating this as
> Phase
> > 1
> > > so
> > > > the contract and the SPI > > primitive > > >>>> can be > > >>>> > >
> > > > reviewed before the larger operation migrations land. > > >>>> > > >
> >
> > > >>>>
> > > > > > Could you please take a look? In particular, I would like to >
> know
> > > > >
> > > > >>>> whether > > >>>> > > the contract captures the consensus so far,
> > or
> > > if
> > > > it needs to > > >>>> address retry > > >>>> > > / external-work
> > > > coordination before we merge this foundation. > > >>>> > > > > >>>>
> > >
> > > > Thanks, > > >>>> > > Prithvi S > > >>>> > > > > >>>> > > On Mon, Aug
> > 17,
> > > > 2026 at 7:40 PM Dmitri Bourlatchkov < > > >>>> dimas@apache.org> > >
> > > >>>>
> > > > > > wrote: > > >>>> > > > > >>>> > > > Hi Robert, > > >>>> > > > > >
> > > >>>> >
> > > > > > I agree that the client-visible operation (e.g. REST API > >
> > > request) >
> > > > > >>>> > > consists > > >>>> > > > of many distinct phases. The
> > database
> > > /
> > > > persistence changes > are > > >>>> just one > > >>>> > > of > > >>>>
> >
> > >
> > > >
> > > > these phases. STS (as an example) is another. My point about > > >>>>
> > > > transactions > > >>>> > > > in JDBC Persistence applies to the former
> > > > (database changes) > > >>>> phase - one > > >>>> > > > retry attempt
> > > there
> > > > should ideally involve exactly one > > >>>> transaction. > > >>>> >
> > >
> > > > >
> > > > >>>> > > > We certainly need logic inside Polaris Servers to to >
> > > > coordinate > > >>>> requests > > >>>> > > to > > >>>> > > > various
> > > > external systems depending on outcomes from previous > > >>>> request
> > > >
> > > > >>>> > > > processing phases. > > >>>> > > > > > >>>> > > > I also
> > agree
> > > > that validation based on database state should be > > >>>> repeated
> > >
> > > > >>>> > > > from scratch if we retry the database changes. > > >>>> >
> >
> > >
> > > >
> > > > > >>>> > > > However, internal RBAC authorization naturally has to
> > > happen >
> > > > > >>>> inside the > > >>>> > > > database update phase
> > > (read-check-write).
> > > > IIRC, that is part > of > > >>>> the > > >>>> > > > implicit
> > consistency
> > > > guarantees, which were discussed during > > early > > >>>> > >
> project
> > >
> > > >
> > > > >>>> > > > months (but never got written down in full clarity, > >
> >>>>
> > > > unfortunately). If > > >>>> > > the > > >>>> > > > Authorization
> check
> > is
> > > > inside that phase for internal RBAC, I > > >>>> suppose it > > >>>> >
> > > >
> > > > will be there for other Authorizers too and that can involve > > >>>>
> > > > external > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly
> > > > expect > > >>>> authorization to be > > >>>> > > > efficient, but I'm
> > not
> > > > sure how quick it can be in practice. > > >>>> There's > > >>>> > > >
> > > > certainly potential for delays in some (perhaps infrequent) > >
> cases.
> > >
> > > >
> > > > >>>> > > > > > >>>> > > > Cheers, > > >>>> > > > Dmitri. > > >>>> >
> > >
> > > > >
> > > > >>>> > > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert
> Stupp
> > <
> > > > snazy@snazy.de> > > >>>> wrote: > > >>>> > > > > > >>>> > > > > Hi
> > > > Dmitri, > > >>>> > > > > > > >>>> > > > > I agree that one logical
> > > > operation needs a consistent view > of > > >>>> the whole > > >>>> >
> >
> > >
> > > >
> > > > backend state. > > >>>> > > > > What worries me is treating one long
> > JDBC
> > > > transaction as the > > >>>> boundary > > >>>> > > of > > >>>> > > > >
> > > that
> > > > operation and then retrying the complete request if the > > >>>>
> commit
> > > > >
> > > > >>>> > > > fails. > > >>>> > > > > > > >>>> > > > > The request can
> > also
> > > > write metadata to object storage or > call > > >>>> STS. > > >>>> > >
> > > >
> > > > Each system can leave us with an outcome whose certainty we > > >>>>
> > > cannot
> > > > > > >>>> > > know. > > >>>> > > > > The database may have committed
> > > before
> > > > the connection was > > lost, > > >>>> an > > >>>> > > > >
> object-store
> > > > write may have succeeded before a timeout, or > STS > > >>>> may have
> > > >
> > > > >>>> > > > > issued credentials before its response was lost. > >
> >>>>
> > >
> > > >
> > > > > > A database rollback cannot undo any of those effects, and > >
> >>>>
> > > > retrying the > > >>>> > > > > whole request may repeat them. > >
> >>>> >
> > > > >
> > > > > > > >>>> > > > > So I think the contract has to separate the
> overall
> > >
> > > > operation > > >>>> from each > > >>>> > > > > attempt to commit it.
> > >
> > > > >>>> > > > > Each database or backend commit attempt should stay
> short.
> > > > >
> > > > >>>> > > > > When we retry, validation and authorization need to use
> > the
> > > >
> > > > > >>>> state for > > >>>> > > > that > > >>>> > > > > new attempt,
> and
> > we
> > > > need clear rules for which external work > > >>>> can be > > >>>> >
> > >
> > > > reused, > > >>>> > > > > repeated, cleaned up, or reconciled after an
> > > > uncertain > > outcome. > > >>>> > > > > > > >>>> > > > > Some
> > contextual
> > > > state could be useful for carrying that > state. > > >>>> > > > > But
> > the
> > > > JDBC connection and transaction should be an > > >>>> implementation
> >
> > >
> > > > >>>> > > > detail > > >>>> > > > > inside it, not the boundary of the
> > > > complete operation. > > >>>> > > > > > > >>>> > > > > Cheers, > >
> >>>>
> > >
> > > >
> > > > > > Robert > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > On Mon,
> > Aug
> > > > 10, 2026 at 11:04 PM Dmitri Bourlatchkov < > > >>>> dimas@apache.org
> > > >
> > > > >>>> > > > > > >>>> > > > > wrote: > > >>>> > > > > > > >>>> > > > >
> >
> > Hi
> > > > All, > > >>>> > > > > > > > >>>> > > > > > The idea of a
> > backend-agnostic
> > > > change-set primitive sounds > > >>>> good. > > >>>> > > > > > > >
> >>>>
> > >
> > > >
> > > > > > > However, I am not sure it is sufficient for all the use > >
> cases
> > > > >
> > > > >>>> we > > >>>> > > > touched > > >>>> > > > > > here. More
> > > specifically,
> > > > the state read by validation code > > is > > >>>> not > > >>>> > > >
> >
> > >
> > > > necessarily reflected in the change set: > > >>>> > > > > > > > >>>>
> >
> > >
> > > >
> > > > > > * Unchanged entities might be considered by validation, > but > >
> > > >>>>
> > > > changed > > >>>> > > in > > >>>> > > > a > > >>>> > > > > > parallel
> > > > request. > > >>>> > > > > > * Even for changed entities the state
> read
> > by
> > > > validation > may > > >>>> differ > > >>>> > > > from > > >>>> > > >
> > >
> > > the
> > > > state being altered. Some validation code talks to the > > >>>>
> > > MetaStore >
> > > > > >>>> > > > > > directly, outside of the data produced by the
> > Resolver.
> > > >
> > > > > >>>> > > > > > > > >>>> > > > > > I do not think Polaris offers any
> > > > explicit mechanisms > (ATM) > > >>>> to ensure > > >>>> > > > > >
> > > > consistency between these reads and subsequent writes. > > >>>> > >
> > >
> > > > >
> > > > > >>>> > > > > > As far as JDBC goes, running an overarching
> > Transaction
> > > >
> > > > > >>>> across all > > >>>> > > > reads > > >>>> > > > > > and writes
> in
> > > the
> > > > same request, at the SERIALIZABLE > > isolation > > >>>> level > >
> >>>>
> > > > >
> > > > in > > >>>> > > > > the > > >>>> > > > > > backing RDBMS could solve
> > the
> > > > problem, I think. > > >>>> > > > > > > > >>>> > > > > > Indeed, such
> a
> > > > transaction might be long to accommodate > > calls > > >>>> made by
> > >
> > > > >>>> > > > > > Polaris to external storage, etc. However, is that a
> > >
> > > > problem? > > >>>> I think > > >>>> > > > the > > >>>> > > > > >
> > > alternative
> > > > is for the JDBC Persistence impl. to > "manually" > > >>>> track all
> >
> > >
> > > > >>>> > > > > reads > > >>>> > > > > > under the same request and redo
> > > them
> > > > in the small > > transaction > > >>>> that > > >>>> > > > > persists
> >
> > >
> > > > >>>> > > > > > the writes. This will also add RDBMS overhead and
> > require
> > > >
> > > > > >>>> complex > > >>>> > > code > > >>>> > > > in > > >>>> > > > > >
> > > > Polaris to handle the data properly. > > >>>> > > > > > > > >>>> > >
> >
> > >
> > > >
> > > > Tracking the request-wide transaction can be done only for > > >>>>
> > JDBC
> > > >
> > > > > >>>> > > without > > >>>> > > > > > leaking "transaction" concepts
> to
> > > the
> > > > NoSQL Persistence, I > > >>>> think. > > >>>> > > NoSQL > > >>>> > >
> >
> > >
> > > >
> > > > will use other mechanisms to ensure read/write > consistency. > >
> >>>>
> > >
> > > >
> > > > > > > > > >>>> > > > > > Connecting to Robert's email (a parallel
> > branch
> > > in
> > > > this > > >>>> discussion), > > >>>> > > > I'd > > >>>> > > > > > like
> > to
> > > > propose this approach: > > >>>> > > > > > > > >>>> > > > > > * Each
> > > request
> > > > establishes a "Data Context" > > >>>> > > > > > - In JDBC the Data
> > > Context
> > > > corresponds to a JDBC > > Connection > > >>>> + Tx > > >>>> > > > >
> > -
> > > In
> > > > NoSQL the Data Context tracks one or more reference > > >>>> hashes
> > >
> > > > >>>> > > > > > * All Persistence access in the same request goes
> > through
> > > >
> > > > > the > > >>>> same > > >>>> > > Data > > >>>> > > > > > Context > >
> > > >>>> >
> > > > > > > > * All Persistence changes are committed once at the end of >
> >
> > > the
> > > > > > >>>> > > request > > >>>> > > > > > - Not all changes have to be
> > > > globally atomic. For > example, > > >>>> changes > > >>>> > > in > >
> > > >>>> >
> > > > > > > > Catalogs A and B do not have to be atomic with respect to > >
> > > >>>>
> > > > each other. > > >>>> > > > We > > >>>> > > > > > can go deeper into
> > this
> > > > later. This is relevant to NoSQL. > > >>>> > > > > > - Transactional
> > > > backends like JDBC can, of course, > choose > > >>>> to make > > >>>>
> > > >
> > > > all > > >>>> > > > > > changes globally atomic. > > >>>> > > > > > *
> A
> > > > commit can fail in two main ways: > > >>>> > > > > > - A retriable
> > > failure
> > > > like an optimistic lock error or > Tx > > >>>> > > > >
> serializability
> > >
> > > >
> > > > >>>> > > > > > error > > >>>> > > > > > - A non-triable logical error
> > > (e.g.
> > > > entity not found) > > >>>> > > > > > * On a retriable error the whole
> > > > request is re-attempted > (as > > >>>> if > > >>>> > > > >
> resubmitted
> > >
> > > >
> > > > >>>> > > > > > by a client) a few times (configurable timeout). > >
> > >>>>
> > > >
> > > > > > > > > > >>>> > > > > > WDYT? > > >>>> > > > > > > > >>>> > > > >
> >
> > > > Cheers, > > >>>> > > > > > Dmitri. > > >>>> > > > > > > > >>>> > > >
> >
> > >
> > > On
> > > > Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré < > > >>>> > >
> > > > jb@nanthrax.net > > >>>> > > > > > > >>>> > > > > > wrote: > > >>>>
> >
> > >
> > > >
> > > > > > > > >>>> > > > > > > Hi all > > >>>> > > > > > > > > >>>> > > >
> > >
> > > > I
> > > > think Robert has a good point. > > >>>> > > > > > > > > >>>> > > > >
> >
> > >
> > > If
> > > > the atomicity guarantee lives only in > BasePersistence, > > >>>>
> then
> > > the
> > > > > > >>>> > > > > > > manager contract can't tell a caller whether the
> > > state
> > > > > it > > >>>> read for > > >>>> > > > > > >
> > > > validation/authorization/credential-vending is the same > > >>>>
> state
> > > that
> > > > > > >>>> > > > > > > eventually commits. That's the actual bug class
> > > behind
> > > > > the > > >>>> JDBC > > >>>> > > > > > > symptoms (not any single
> > > operation
> > > > being non-atomic, but > > the > > >>>> > > contract > > >>>> > > > >
> >
> > >
> > > > being silent about it. Every operation-specific fix > (like > > >>>>
> > > #4939,
> > > > > > >>>> > > > #5035 > > >>>> > > > > > > or #5095) re-answers this
> > > > question locally and it keeps > > >>>> recurring. > > >>>> > > So > >
> > > >>>>
> > > > > > > > > > the deliverable should start with a written consistency
> > >
> > > > >>>> contract at > > >>>> > > > > > > the manager level. > > >>>> >
> > >
> > > > >
> > > > > > > >>>> > > > > > > I'm not sure migrating everyone to > > >>>> >
> >
> > > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > is a good
> idea.
> > It
> > > > would tie the logical change set to a > > DB > > >>>> > > > > > >
> > > > transaction spanning the REST request. It means: > > >>>> > > > > >
> > -
> > > it
> > > > holds a durable transaction open across slow > external > > >>>> work
> > > >
> > > > >>>> > > > > > > (credential vending, OPA, Ranger, ...) > > >>>> > >
> >
> > >
> > > >
> > > > > - it doesn't map to NoSQL > > >>>> > > > > > > - It wraps
> single-row
> > > > updates in runWiithinTransaction, > > >>>> which is a > > >>>> > > >
> >
> > >
> > > > overhead > > >>>> > > > > > > > > >>>> > > > > > > So, I think the
> > > > transactional manager isn't a portable > > >>>> target. It's > >
> >>>> >
> > > > >
> > > > > > > "only" one backend's strategy. > > >>>> > > > > > > > > >>>> >
> >
> > >
> > > >
> > > > > > I think Privthi's approach is right. A backend-agnostic > > >>>>
> > > > change-set > > >>>> > > > > > > primitive with a documented fallback
> is
> > > the
> > > > correct > shape. > > >>>> There is > > >>>> > > > > > > one caveat:
> > > without
> > > > carrying the original entitiy (not > > just > > >>>> the new > >
> >>>> >
> > > > >
> > > > > > > one) the change set can't express optimistic > concurrency. > >
> > > >>>>
> > > > > > > > > > > > >>>> > > > > > > I propose the following multi-steps
> > > > approach: > > >>>> > > > > > > 1. We write the manager-level
> > consistency
> > > > contract as > > >>>> Robert asked. > > >>>> > > > It > > >>>> > > > >
> > > >
> > > > should include the explicit statement that a logical > > change > >
> > >>>>
> > > > set is > > >>>> > > > not > > >>>> > > > > > > a request-scope DB
> > > > transaction. > > >>>> > > > > > > 2. We make Compare And Swap
> > > > (optimistic-concurrency > > pattern) > > >>>> > > baseline > > >>>>
> > >
> > > > >
> > > > > > first-class in EntityMutation before merging the SPI > > >>>> >
> > >
> > > > >
> > > > > 3. Then we refactor > createCatalog/dropEntity/renameEntity. > >
> >>>>
> > > > >
> > > > > > > > > > >>>> > > > > > > Thoughts? > > >>>> > > > > > > > > >>>>
> >
> > >
> > > >
> > > > > > > Regards > > >>>> > > > > > > JB > > >>>> > > > > > > > > >>>>
> > >
> > > > >
> > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp < > > >>>>
> > > snazy@snazy.de>
> > > > > > >>>> > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > Hi
> > > all, >
> > > > > >>>> > > > > > > > > > >>>> > > > > > > > Yufei's clarification
> about
> > > > where the atomicity > > guarantee > > >>>> is > > >>>> > > > defined
> >
> > >
> > > > >>>> > > > > > > seems > > >>>> > > > > > > > important. > > >>>> >
> > >
> > > > >
> > > > > > If it is a property of the lower-level BasePersistence > > >>>>
> > > > contract > > >>>> > > > > rather > > >>>> > > > > > > than > > >>>>
> > >
> > > > >
> > > > > > > the general PolarisMetaStoreManager contract, the > > general
> > >
> > > > >>>> > > contract > > >>>> > > > > does > > >>>> > > > > > > not > >
> > > >>>> >
> > > > > > > > > > tell callers whether the state used for validation, > >
> > >>>>
> > > >
> > > > > authorization, > > >>>> > > > or > > >>>> > > > > > > > credential
> > > > vending is consistent with the change that > > >>>> eventually > >
> >>>>
> > > > >
> > > > > > > commits. > > >>>> > > > > > > > > > >>>> > > > > > > > The
> > current
> > > > PRs suggest that operation-specific > > >>>> multi-object > > >>>> >
> >
> > >
> > > > methods > > >>>> > > > > > can > > >>>> > > > > > > > fix individual
> > > cases,
> > > > while leaving the same contract > > >>>> question to > > >>>> > > > >
> > > recur
> > > > > > >>>> > > > > > > for > > >>>> > > > > > > > each new case. > >
> >>>>
> > > > >
> > > > > > > > > > > >>>> > > > > > > > I would also avoid defining a
> logical
> > > > change set as a > > >>>> database > > >>>> > > > > > > transaction >
> >
> > > >>>>
> > > > > > > > > > > around an entire REST request. > > >>>> > > > > > > >
> > That
> > > > would tie the contract to the backend and > database, > > >>>> and >
> >
> > > >>>>
> > > > > > could > > >>>> > > > > keep > > >>>> > > > > > > the > > >>>> >
> > >
> > > > >
> > > > > > durable attempt open across slow external work. > > >>>> > > > >
> >
> > >
> > > >
> > > > > > >>>> > > > > > > > So is the choice really between the two
> current
> > >
> > > > manager > > >>>> > > > > > implementations, > > >>>> > > > > > > or
> > >
> > > > >>>> > > > > > > > do we first need to revisit the boundary and >
> > > > guarantees > > >>>> exposed > > >>>> > > to > > >>>> > > > > > their
> >
> > >
> > > > >>>> > > > > > > > callers? > > >>>> > > > > > > > > > >>>> > > > > >
> > > >
> > > > Cheers, > > >>>> > > > > > > > Robert > > >>>> > > > > > > > > >
> >>>> >
> > > > >
> > > > > > > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM
> Dmitri
> > > > Bourlatchkov < > > >>>> > > > > dimas@apache.org > > >>>> > > > > >
> >
> > >
> > > >
> > > > >>>> > > > > > > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > >
> > > >
> > > > Hi Yufei, > > >>>> > > > > > > > > > > >>>> > > > > > > > > Thanks
> for
> > > the
> > > > link. I stand corrected. The "one > > atomic > > >>>> change > >
> >>>> >
> > > > >
> > > > per > > >>>> > > > > > > method" > > >>>> > > > > > > > > contract as
> > > > defined in javadoc does apply only to > > >>>> > > BasePersistence >
> >
> > > >>>>
> > > > > > > > > > > > and AtomicOperationMetaStoreManager (which
> delegates >
> > >
> > > to
> > > > > > >>>> > > > > > > BasePersistence). > > >>>> > > > > > > > > > >
> > >>>>
> > > >
> > > > > > > > > > > Note that all non-test Persistence implementations > in
> > > >
> > > > >>>> the > > >>>> > > Polaris > > >>>> > > > > > > codebase > > >>>>
> >
> > >
> > > >
> > > > > > > > > extend those classes (which is probably why I was > > >>>>
> > > > confused > > >>>> > > about > > >>>> > > > > > > atomicity > > >>>>
> > >
> > > > >
> > > > > > > > expectations). > > >>>> > > > > > > > > > > >>>> > > > > > >
> >
> > >
> > > > However, this creates a gap in the Persistence SPI > > >>>> > >
> > > > specification. > > >>>> > > > > If > > >>>> > > > > > > > > other
> > > > PolarisMetaStoreManager implementations do not > > >>>> have to > >
> > >>>>
> > > >
> > > > > > comply > > >>>> > > > > > > with > > >>>> > > > > > > > > this
> > > > principle, it will create a conceptual > difficulty > > >>>> at call
> >
> > >
> > > > >>>> > > > > sites. > > >>>> > > > > > > How > > >>>> > > > > > > > >
> > can
> > > > PolarisMetaStoreManager callers reason about > > >>>> consistency > >
> > > >>>>
> > > > > > and > > >>>> > > > > > > durability > > >>>> > > > > > > > >
> > > behaviours
> > > > in general? > > >>>> > > > > > > > > > > >>>> > > > > > > > > I
> believe
> > > we
> > > > need to address that as part of this > > >>>> discussion. > > >>>> >
> >
> > >
> > > >
> > > > > > > > > > >>>> > > > > > > > > Cheers, > > >>>> > > > > > > > >
> > > Dmitri. >
> > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > On Thu, Jul 16, 2026
> at
> > > > 9:29 PM Yufei Gu < > > >>>> flyrain000@gmail.com > > >>>> > > > > >
> > >>>>
> > > > > > > > > wrote: > > >>>> > > > > > > > > > > >>>> > > > > > > > > >
> >
> > > The
> > > > MetaStore SPI is currently defined with the > > >>>> idea that > >
> >>>>
> > > > >
> > > > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > >
> >>>>
> > >
> > > >
> > > > > > > > > > > means one atomic change. > > >>>> > > > > > > > > > > >
> > > >>>>
> > > > > > > > > > > > > If the "MetaStore SPI" refers to the interface > >
> > > >>>> >
> > > > > > > > > PolarisMetaStoreManager, I > > >>>> > > > > > > > > > don't
> > > think
> > > > we've ever state each method to be > > >>>> atomic. We did > > >>>>
> > >
> > > > >
> > > > > clarify > > >>>> > > > > > > > > > atomicity[1] in the interface
> > > > BasePersistence > > though. > > >>>> > > > > > > > > > > > >>>> > >
> > >
> > > > >
> > > > > > > > > >>>> > > > > > > > > > 1. > > >>>> > > > > > > > > > > >
> >>>>
> > > > >
> > > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>>
> >
> > >
> > > >
> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > >
> > > >
> > >
> > https://github.com/apache/polaris/blob/e9039e12003a13e783b51
> 30a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/
> polaris/core/persistence/BasePersistence.java#L48
> > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > >
> >
> > >
> > > >
> > > > > > > > > >>>> > > > > > > > > > Yufei > > >>>> > > > > > > > > > > >
> > > >>>>
> > > > > > > > > > > > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at
> > > > 11:27 AM Dmitri > > Bourlatchkov < > > >>>> > > > > > >
> > dimas@apache.org
> > > >
> > > > > > >>>> > > > > > > > > > wrote: > > >>>> > > > > > > > > > > >
> >>>> >
> > > > >
> > > > > > > > > > > Hi Yufei, > > >>>> > > > > > > > > > > > > >>>> > > >
> > >
> > > > >
> > > > > > > I agree that JDBC transactions must be handled > > more > >
> >>>>
> > >
> > > >
> > > > > explicitly. > > >>>> > > > > > > > > However, > > >>>> > > > > > >
> >
> > >
> > > >
> > > > > I'm not sure that simply moving to > > >>>> > > > > > >
> > > > TransactionalMetaStoreManagerImpl is > > >>>> > > > > > > > > > >
> > > > sufficient. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > >
> > The
> > > > MetaStore SPI is currently defined with the > > >>>> idea that > >
> >>>>
> > > > >
> > > > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > >
> >>>>
> > >
> > > >
> > > > > > > > > > > > means one atomic change [1]. The "transactional" > >
> > >>>>
> > > > MetaStore > > >>>> > > > > impl. > > >>>> > > > > > is > > >>>> > >
> > >
> > > > >
> > > > > > but a > > >>>> > > > > > > > > > > sub-case of that. It cannot
> > alter
> > > > the high-level > > >>>> contract. > > >>>> > > > > > > > > > > > >
> >>>>
> > > > >
> > > > > > > > > > > > We could add SPI methods having multiple object > >
> > >>>>
> > > > parameters > > >>>> > > to > > >>>> > > > > > > represent > > >>>> >
> >
> > >
> > > >
> > > > > > > > > > grouped changes, but I am not sure it will be a > > >>>>
> > > sound
> > > > > > >>>> > > design. > > >>>> > > > > > This > > >>>> > > > > > >
> will
> > >
> > > >
> > > > >>>> > > > > > > > > > > bloat the interface surfaces and require
> extra
> > > > >
> > > > >>>> impl. effort > > >>>> > > > for > > >>>> > > > > > > each > >
> >>>>
> > > > >
> > > > > > > > > > > > backend type. More importantly, adding multi-arg > >
> > >>>>
> > > > change > > >>>> > > > methods > > >>>> > > > > > > still > > >>>> >
> > >
> > > > >
> > > > > > > > won't > > >>>> > > > > > > > > > > address the problem of
> reads
> > > > being consistent > with > > >>>> writes, > > >>>> > > > > > because
> > >
> > > > >>>> > > > > > > each > > >>>> > > > > > > > > > > method call will
> > still
> > > > be independent regarding > > the > > >>>> data > > >>>> > > > stored
> >
> > >
> > > > >>>> > > > > > in > > >>>> > > > > > > the > > >>>> > > > > > > > >
> > >
> > > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > I
> > > tend
> > > > to think we need to introduce a "change > > set" > > >>>> or > >
> >>>> >
> > > > >
> > > > "atomic > > >>>> > > > > > > batch" > > >>>> > > > > > > > > > >
> > concept
> > > to
> > > > core Persistence and associate each > > REST > > >>>> API > > >>>> >
> >
> > >
> > > > request > > >>>> > > > > > > with > > >>>> > > > > > > > > one > >
> >>>>
> > > > >
> > > > > > > > > > > > such change set, which will be committed (or > >
> rolled
> > > > >
> > > > >>>> back) > > >>>> > > at > > >>>> > > > > the > > >>>> > > > > > >
> > end
> > > of
> > > > > > >>>> > > > > > > > > > the > > >>>> > > > > > > > > > > request.
> I
> > > > believe Ayush mentioned a similar > > >>>> concept in PR > > >>>> >
> > >
> > > > 4939 > > >>>> > > > > > > [2]. In > > >>>> > > > > > > > > > > JDBC
> > each
> > > > change set will naturally be > associated > > >>>> with an > > >>>> >
> > > >
> > > > RDBMS > > >>>> > > > > > > > > > > transaction. In NoSQL persistence,
> > > each
> > > > atomic > > >>>> change set > > >>>> > > > will > > >>>> > > > > be >
> >
> > > >>>>
> > > > > > > > > > > > > > associated with one CAS operation on the > >
> > > underlying
> > > > > > >>>> > > database. > > >>>> > > > > > > > > > > > > >>>> > > > >
> >
> > >
> > > >
> > > > > > > [1] > > >>>> > > > > > > > > >>>>
> > > > https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > >
> > > >>>>
> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > [2] > > >>>> > > >
> > >
> > > > >
> > > > > >>>>
> > > https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > Cheers, > >
> > > >>>>
> > > > > > > > > > > > > > Dmitri. > > >>>> > > > > > > > > > > > > >>>> >
> > >
> > > > >
> > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < > > >>>> > > >
> > > > flyrain000@gmail.com > > >>>> > > > > > > > >>>> > > > > > > wrote:
> >
> > >
> > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > Thanks for
> > > raising
> > > > this, Dmitri. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > >
> > >
> > > > >
> > > > These are valid concerns, and they were > already > > >>>> > >
> > > recognized >
> > > > > >>>> > > > > when > > >>>> > > > > > > we > > >>>> > > > > > > > >
> >
> > >
> > > >
> > > > introduced JDBC persistence to Polaris. At > that > > >>>> time, we
> > >
> > > > >>>> > > > > chose > > >>>> > > > > > > to use > > >>>> > > > > > >
> > >
> > > > >
> > > > > AtomicOperationMetaStoreManager for the JDBC > due > > >>>> to
> the >
> > >
> > > > >>>> > > > > > > simplicity. I > > >>>> > > > > > > > > > > think > >
> > > >>>>
> > > > > > > > > > > > > > > most of the issues mentioned here can already >
> > be
> > > >
> > > > > >>>> > > addressed > > >>>> > > > by > > >>>> > > > > > > > > > > >
> > > > TransactionalMetaStoreManagerImpl. > > >>>> > > > > > > > > > > > >
> >
> > > >>>>
> > > > > > > > > > > > > > > For example, rename is already wrapped in a > >
> > > >>>>
> > > > transaction > > >>>> > > in > > >>>> > > > > > > > > > > >
> > > > TransactionalMetaStoreManagerImpl [1]. > > Similarly, > > >>>>
> catalog
> > >
> > > >
> > > > >>>> > > > > > > creation, > > >>>> > > > > > > > > > which > > >>>>
> >
> > >
> > > >
> > > > > > > > > > > > involves reading and creating multiple > objects, > >
> > > >>>>
> > > > is also > > >>>> > > > > > executed > > >>>> > > > > > > > > >
> within a
> > > > >
> > > > >>>> > > > > > > > > > > > transaction [2]. > > >>>> > > > > > > > >
> >
> > >
> > > >
> > > > > > >>>> > > > > > > > > > > > I see two possible directions: > >
> >>>>
> > >
> > > >
> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > >
> >
> > >
> > > >
> > > > > > > > > > >>>> > > > > > > > > > > > Modify
> > > > AtomicOperationMetaStoreManager > > >>>> together with > > >>>> > >
> the
> > > > >
> > > > >>>> > > > > > > > > persistence > > >>>> > > > > > > > > > > >
> > backends
> > > > (such as JDBC) to provide the > > required > > >>>> > > > >
> consistency
> > > > >
> > > > >>>> > > > > > > > > > guarantees > > >>>> > > > > > > > > > > > for
> >
> > >
> > > > >>>> > > > > > > > > > > > specific operations, similar to what > >
> > >>>>
> > > >
> > > > > > > > > > > > TransactionalMetaStoreManagerImpl > > >>>> > > > >
> > >
> > >
> > > >
> > > > > > > does. > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > >
> >
> > >
> > > >
> > > > > > > >>>> > > > > > > > > > > > Migrate the persistence backends
> (such
> > > as
> > > > > > >>>> JDBC) to use > > >>>> > > > > > > > > > > >
> > > > TransactionalMetaStoreManagerImpl directly. > > We > > >>>> may > >
> > >>>>
> > > >
> > > > > have > > >>>> > > > to > > >>>> > > > > > > deal > > >>>> > > > >
> > >
> > > > >
> > > > with > > >>>> > > > > > > > > > > > transactional semantic mismatches
> > > > across > > >>>> different > > >>>> > > > > > persistence > > >>>> >
> > >
> > > > >
> > > > > > > > > backends. > > >>>> > > > > > > > > > > > For example, we
> > would
> > > > likely avoid using > > JDBC's > > >>>> > > > > > > > > >
> > > > `runWithinTransaction` > > >>>> > > > > > > > > > > > for > > >>>> >
> >
> > >
> > > >
> > > > > > > > > > > single row updates, which adds additional > > >>>>
> > overhead
> > > > and > > >>>> > > > > > > complexity > > >>>> > > > > > > > > > > >
> > > without
> > > > > > >>>> > > > > > > > > > > > benefits. > > >>>> > > > > > > > > >
> > >
> > > > >
> > > > >>>> > > > > > > > > > > > References: > > >>>> > > > > > > > > > > >
> > > >
> > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > >
> >>>> >
> > > > >
> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > >
> > >
> > > > >
> > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> >
> >
> > >
> > > >
> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > >
> > > >>>> >
> > > > > > > > >>>> > > > > >>>> > > >
> > > >
> > >
> > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f
> 78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/
> polaris/core/persistence/transactional/TransactionalMetaStor
> eManagerImpl.java#L1286
> > > > > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > > > > >
> > > >>>> >
> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > >
> > >
> > > > >
> > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > >
> >>>>
> > >
> > > >
> > > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > >
> >
> > >
> > > > >>>> > > > > > >>>> > > > > >>>> > > >
> > > >
> > >
> > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f
> 78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/
> polaris/core/persistence/transactional/TransactionalMetaStor
> eManagerImpl.java#L965
> > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > Yufei
> > >
> > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> >
> >
> > >
> > > >
> > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri > > >>>>
> > > Bourlatchkov
> > > > < > > >>>> > > > > > > > > dimas@apache.org> > > >>>> > > > > > > >
> >
> > >
> > > >
> > > > > wrote: > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > >
> >
> > >
> > > Hi
> > > > all, > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > >
> >
> > > > Ayush and Prithvi recently contributed a > > couple > > >>>> of > >
> > >>>>
> > > >
> > > > > > > > interesting > > >>>> > > > > > > PRs: > > >>>> > > > > > > >
> >
> > >
> > > >
> > > > > > [4939], [5035]. > > >>>> > > > > > > > > > > > > > > >>>> > > >
> > >
> > > > >
> > > > > > > > > It looks like people are starting to > encounter > > >>>> >
> > > >
> > > > consistency > > >>>> > > > > > > issues > > >>>> > > > > > > > > in
> > >
> > > > >>>> > > > > > > > > > > > > JDBC persistence. > > >>>> > > > > > >
> > >
> > > > >
> > > > > > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight
> > > into
> > > > the > > >>>> underlying > > >>>> > > > > issues. > > >>>> > > > > > >
> > > They
> > > > > > >>>> > > > > > > > > > offer > > >>>> > > > > > > > > > > > >
> > > > incremental fixes that can work. However, I > > >>>> believe it > >
> > >>>>
> > > >
> > > > > is > > >>>> > > > > > time > > >>>> > > > > > > for > > >>>> > > >
> >
> > >
> > > >
> > > > > > > the > > >>>> > > > > > > > > > > > > Polaris community to
> review
> > > and
> > > > improve this > > >>>> area of > > >>>> > > the > > >>>> > > > > > >
> > > > codebase > > >>>> > > > > > > > > > > > > holistically. > > >>>> > >
> >
> > >
> > > >
> > > > > > > > > > > > > >>>> > > > > > > > > > > > > By this, I mean
> finding
> > a
> > > > solution that can > be > > >>>> applied > > >>>> > > to > > >>>> > >
> >
> > >
> > > > all > > >>>> > > > > > > > > > > > > persistence backends (in-memory,
> > > JDBC,
> > > > > NoSQL) > > >>>> and > > >>>> > > > addresses > > >>>> > > > > > >
> > > these >
> > > > > >>>> > > > > > > > > > > > > aspects: > > >>>> > > > > > > > > > >
> >
> > >
> > > >
> > > > > >>>> > > > > > > > > > > > > * Supporting concurrent and
> consistent >
> > > > changes > > >>>> where > > >>>> > > the > > >>>> > > > > > > service
> >
> > >
> > > > >>>> > > > > > > > > > reads > > >>>> > > > > > > > > > > > > and
> > > validates
> > > > current catalog state, then > > >>>> commits a > > >>>> > > > >
> change
> > >
> > > >
> > > > >>>> > > > > > > (e.g. > > >>>> > > > > > > > > > > > > name clashes
> > > during
> > > > renames). > > >>>> > > > > > > > > > > > > * Supporting consistent
> but
> > > > independent > > changes > > >>>> to RBAC > > >>>> > > > > > grants >
> >
> > > >>>>
> > > > > > > > > > and > > >>>> > > > > > > > > > > > > MetaStore entities.
> > This
> > > > independence is > > >>>> needed to > > >>>> > > > > support > > >>>>
> >
> > >
> > > >
> > > > > > > > > > > > > external authorizers like OPA and Ranger. > > >>>>
> >
> > >
> > > >
> > > > > > > > > > > > > * Supporting atomic changes across multiple > >
> >>>>
> > > > similar > > >>>> > > > > entities. > > >>>> > > > > > > > > > > > > *
> > > > Supporting authorization-based filtering > of > > >>>> list > > >>>>
> >
> > >
> > > >
> > > > > operations > > >>>> > > > > > > (cf. > > >>>> > > > > > > > > > >
> > >
> > > > [4831]). > > >>>> > > > > > > > > > > > > * Supporting
> > credential-vending
> > > > decisions > that > > >>>> are > > >>>> > > rooted > > >>>> > > > > in
> > > >
> > > > >>>> > > > > > > the > > >>>> > > > > > > > > > > > > exact state of
> > the
> > > > catalog. > > >>>> > > > > > > > > > > > > * Supporting server-side
> > > retries
> > > > for > transient > > >>>> > > > persistence > > >>>> > > > > > >
> > failures
> > > >
> > > > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability
> > failures).
> > > >
> > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > >
> Please
> > > > share your comments and ideas. > > >>>> > > > > > > > > > > > > > >
> > >>>>
> > > >
> > > > > > > > > > > > > > > [4831] > > >>>>
> > > > https://github.com/apache/polaris/pull/4831 > > >>>> > > > > > > > >
> > > >
> > > > > > > > >>>> > > > > > > > > > > > > [4939] > > >>>>
> > > > https://github.com/apache/polaris/pull/4939 > > >>>> > > > > > > > >
> > > >
> > > > > > > > >>>> > > > > > > > > > > > > [5035] > > >>>>
> > > > https://github.com/apache/polaris/pull/5035 > > >>>> > > > > > > > >
> > > >
> > > > > > > > >>>> > > > > > > > > > > > > Thanks, > > >>>> > > > > > > >
> > >
> > > > >
> > > > > Dmitri > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > >
> >
> > >
> > > >
> > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > >
> >
> > >
> > > >
> > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > >
> >>>> >
> > > > >
> > > > > > >>>> > > > > >>>> > > >>> > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi guys,
Thanks a lot EJ for the updated PoC and testing it across multiple backends.
At a high level, I really like the direction and separation of concerns
here:
- Having a business logic centralized in the shared Manager rathe than
re-implemented across backends avoids logic drift and fixes the maintenance
burden we have today. I remember it was my call when we had Community
Meetings were we discussed OpenJPA -> JDBC and later NoSQL.
- Keeping the storage adapter contract focused on basic storage operations
without "coupling" in table/catalog concepts is clean and portable (I have
the feeling it was discussed while back as well :)).
- Ensuring STS and object storage calls stay decoupled from the database
transaction boundary is essential.
I will take a new look on your branch, we probably need a couple of weeks
or so (several of us are travelling to CoC).
Personally, I'm interested in closely examining the failure handling around
uncertain commits.
Thanks!
Regards
JB
On Thu, Oct 8, 2026 at 5:33 AM Dmitri Bourlatchkov <di...@apache.org> wrote:
> Hi EJ,
>
> Thanks for sharing the PoC!
>
> I'll try to explore that code, but cannot promise a quick turnaround :)
> Reviewing this may take me a couple of weeks (due to other priorities).
>
> Cheers,
> Dmitri.
>
> On Mon, Oct 5, 2026 at 4:47 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > One detail I omitted: the PoC also includes FoundationDB and Spanner
> > adapters alongside JDBC. They reuse the same business-aware Manager. The
> > adapters implement generic storage operations without table, catalog or
> > grant rules.
> >
> > This exercises both goals: addressing consistency problems such as
> > concurrent creation at overlapping table locations, and reusing that
> logic
> > across databases. The concurrency tests have run on FoundationDB,
> > PostgreSQL and CockroachDB. Spanner coverage currently uses the emulator,
> > with three concurrency tests skipped.
> >
> > These are targeted feasibility checks. Production validation and
> > integration with Polaris’s existing NoSQL backend remain follow-up work.
> >
> > -ej
> >
> > On Mon, Oct 5, 2026 at 1:36 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> > > Hi Dmitri, all,
> > >
> > > Two requests can both check for overlapping table locations, see none,
> > and
> > > then create conflicting tables. We need the checks and the resulting
> > writes
> > > to be protected together.
> > >
> > > My proposal is to implement these rules once in shared Polaris code
> (the
> > > Manager). This business-aware layer checks them in the transaction that
> > > creates the table. Database adapters would implement a
> business-agnostic
> > > interface (Primitives) for generic reads, writes and commit. They would
> > not
> > > implement table, catalog or grant rules, so adding a database reuses
> the
> > > same Polaris behavior.
> > >
> > > This PoC assumes the database itself can protect the relevant reads and
> > > writes in one serializable transaction. Calls to object storage or
> > > credential services happen outside it.
> > >
> > > The Java PoC
> > > <
> >
> https://github.com/flyingImer/polaris/tree/prototype/durable-java-primitives/persistence/primitives
> > >
> > > reuses the existing transactional Manager and tests selected races,
> > > authorization retries and uncertain commits. It uses a temporary data
> > > layout and reads more data than we'd want in production. Existing NoSQL
> > > integration, production tuning and migration remain follow-up work.
> > >
> > > I'm not proposing an additional request-wide Persistence Session yet.
> > > Existing business-aware request code still owns authorization, external
> > > work and retries.
> > >
> > > The PoC allows an already-authorized metadata change to finish. A fresh
> > > request retry checks authorization again. An uncertain commit stops
> > > automatic retries and retains files that may already be referenced.
> These
> > > policies still need agreement.
> > >
> > > Would this separation be a useful starting point?
> > >
> > > -ej
> > >
> > >
> > > On Tue, 22 Sep 2026 19:08:49 -0400, Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > wrote:
> > >
> > > Hi EJ, > 1. Which backend assumptions should we design for? [...] I
> know
> > I
> > > used the term "transaction" a lot during the call, but the real
> challenge
> > > is about establishing an SPI with clear contracts that could be
> > implemented
> > > by JDBC and NoSQL persistence using the database features natural to
> each
> > > model. I am fairly confident that a JDBC impl. is possible with
> > > Serializable transaction. We should be able to solve all the problems
> > > raised in this thread. However, that will make in-memory caching
> > impossible
> > > because the database needs to see all the reads (and writes) in a Tx to
> > be
> > > able to ensure serializable guarantees. If a cache prevents a read from
> > > hitting the database, that read (e.g. storage config) will not be a
> > factor
> > > in serializability. This connects to my earlier question about the
> > > in-memory cache and JDBC (title: Usefulness of InMemoryEntityCache) Are
> > > different approaches possible with JDBC? Probably. What is involved?
> This
> > > depends on the SPI contract, I think. > 2. How do we protect the data
> > used
> > > by validation? [...] The point mentioned by Robert previously IIRC, is
> > that
> > > we need to ground all runtime decisions to a particular state of the
> > > catalog. If at commit time the state of the catalog in the database
> > shifts
> > > from what was used during validation, the request has to restart from
> > > scratch. Current JDBC Persistence does this only for relationships
> > between
> > > entities and grants and between versions of the same entity.
> > Relationships
> > > between table locations and storage config are not actualized for the
> > > purpose of state consistency tracking. From my POV we need to begin by
> > > formulating this Persistence state in the Persistence SPI terms. Then,
> > > think how we can implement that for JDBC. NoSQL Persistence already
> has a
> > > per-catalog state (not granular), so wiring the new SPI to NoSQL should
> > not
> > > be too hard, I hope. The state in the SPI does not have to be global to
> > the
> > > catalog (as in NoSQL), but it needs to be able to express the idea of
> > which
> > > elements were engaged during request processing so serve as input to
> > > Persistence impl. for cosistency guarantees. If Persistence caches data
> > in
> > > memory (for example), it will have to do so in a way that does not
> > violate
> > > consistency expectations. From my POV we could begin by defining a
> > > "Persistence Session" concept. Each request is 1:1 with a session. All
> > > reads and writes go through the session. The next step would be the
> JDBC
> > > impl., which probably needs some brainstorming... but I wonder if we
> > could
> > > reach a consensus at a high level first. WDYT? Thanks, Dmitri. On Tue,
> > Sep
> > > 22, 2026 at 5:55 PM EJ Wang <ej...@gmail.com> wrote: >
> > > Thanks for the recap, Dmitri. I'd like to follow up on the three points
> > I >
> > > raised during the sync. We didn't have much time to discuss them, so
> it >
> > > would help to confirm these before choosing an SPI design. > > 1. Which
> > > backend assumptions should we design for? > > Within one Polaris server
> > > instance, do we assume entities, grants and > secrets all use the same
> > > database backend, or must we support mixing > backends across them? >
> > Must
> > > every supported persistence implementation provide transactional >
> > > guarantees? Does the database itself need to provide transactions, or
> > can >
> > > Polaris build that capability on top of simpler database operations? >
> >
> > > I'm happy to work with a homogeneous assumption if that's the intended
> >
> > > scope. Existing NoSQL illustrates the second question: it can publish >
> > > changes to multiple objects under one reference using a single
> > conditional
> > > > pointer update. > > 2. How do we protect the data used by validation?
> > > >
> > > Your storage-configuration and location-overlap examples make the
> > problem >
> > > concrete. Consider these races: > > A table creation passes validation
> > > against the catalog's allowed locations, > then another request changes
> > > that configuration before the table is > committed. > Two
> table-creation
> > > requests each check for overlapping locations, both see > no overlap,
> and
> > > then create tables at conflicting locations. > > Checking only the
> record
> > > being written cannot cover these dependencies. The > first also depends
> > on
> > > the catalog configuration. The second depends on the > absence of
> another
> > > overlapping table. > > Could the persistence layer protect these reads
> > > together with the writes, > using backend-internal versions/tokens or
> > > database isolation? Shared code > would still need to identify the
> > required
> > > reads and checks. Could we keep > the mechanism for protecting them
> > inside
> > > each backend? > > 3. Business rules are duplicated across backend
> > > implementations > > Today, the TreeMap > < >
> > >
> >
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481
> > > > > > and JDBC > < >
> > >
> >
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046
> > > > > > implementations each load a secret, check its principal, apply
> the
> > >
> > > rotation/reset steps and write it back. > > Changing these business
> rules
> > > therefore requires keeping multiple > implementations in sync. A fix
> > > applied to one backend can be missed in > another, causing the same
> > Polaris
> > > operation to behave differently depending > on the database. > > My
> > > proposal is to implement each business workflow once, against a common
> >
> > > set of storage operations with defined consistency and failure
> > semantics. >
> > > The shared workflow should not need to know whether those operations
> are
> > >
> > > implemented through JDBC transactions, NoSQL reference CAS or another >
> > > mechanism. Each backend would implement the same storage contract,
> > keeping
> > > > its execution details internal. A new database backend could then
> reuse
> > > the > existing business workflows. > > Do these seem like reasonable
> > > starting points? We can then evaluate the SPI > against the original
> > > failure cases on both JDBC and existing NoSQL. > > -ej > > On Fri, Sep
> > 18,
> > > 2026 at 12:10 PM Dmitri Bourlatchkov <di...@apache.org> > wrote: > > >
> > > Side note: I just came across [5541], which shows a case where a failed
> > > >
> > > request leaves persisted side effects. > > > > This can be seen as a
> > coding
> > > mistake, of course. However, this is about > > executing one request
> as a
> > > single, atomic unit. This is something that > > probably needs
> > foundational
> > > support and coding patterns in Polaris code. > > > > Cheers, > >
> Dmitri.
> > >
> > > > [5541] https://github.com/apache/polaris/pull/5541 > > > > On Fri,
> Sep
> > > 18, 2026 at 3:02 PM Dmitri Bourlatchkov <di...@apache.org> > > wrote:
> >
> > >
> > > > > > Hi All, > > > > > > Here's a recap of the Community Sync
> discussion
> > > as I remember it. I'm > > sure > > > my recollection is not complete,
> so
> > > please add your thoughts to this > > thread. > > > > > > * Polaris
> > > traditionally relies on entity version number to ensure > > >
> consistency
> > > of changes. > > > > > > This works well for internal RBAC grants, which
> > > also have version > numbers > > > cross-referenced at the Persistence
> > > layer. > > > > > > I do not think this works in more general cases like
> > > validating > locations > > > wrt catalog-level storage configuration. >
> > > >
> > > > > > Another difficult case involves location overlaps between
> > concurrent
> > > > > table > > > creations. > > > > > > * We talked about how
> Persistence
> > > calls related to RDBMS transactions > (in > > > the JDBC persistence
> > case).
> > > > > > > > > There were some concerns raised about whether we need to
> > bind >
> > > Persistence > > > API to transactions explicitly. > > > > > > I think
> it
> > is
> > > a valid concern. At the same time, the Persistence API > > > should be
> > > clear about consistency guarantees across all backend > > >
> > > implementations. I think this needs more attention now that the matter
> >
> > of
> > > > > > milti-entioty changes became prominent in [5035]. > > > > > > I
> > > personally believe that RDBMS-based Persistence in Polaris has to use >
> > > >
> > > Serializable transaction isolation. Consequently, the core code needs >
> > to
> > > > > > provide mechanisms to allow the Persistence implementation to
> > > properly > > > connect reads and writes from API requests to JDBC
> > > transactions... > > whether > > > we use "transation" as an explicit
> > terms
> > > in the SPI or not. > > > > > > Note that currently, the same API
> request
> > > performs reads and writes in > > > _separate_ RDBMS transactions (even
> > > separate connections). > > > > > > * We did not discuss Persistence SPI
> > > implications in the call, but > > > currently each Persistence call is
> > > assumed to be one distinct and > atomic > > > change (please correct me
> > if
> > > I'm wrong). > > > > > > Consequently, supporting new use cases involves
> > > introducing new SPI > > > methods with many parameters. This can be
> seen
> > in
> > > PR [5035]. This tends > > to > > > overcomplicate the SPI. > > > > > >
> > > Ideally, I think the SPI should be simplified to deal with multi-entity
> > > >
> > > > and single-entity changes in a coherent manner to avoid any ambiguity
> > > >
> > > about > > > which method the caller should use. > > > > > > [5035]
> > > https://github.com/apache/polaris/pull/5035 > > > > > > Cheers, > > >
> > > Dmitri. > > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri
> Bourlatchkov
> > <
> > > dimas@apache.org > > > > > wrote: > > > > > >> Heads up: This
> discussion
> > > is on the Community Sync call agenda for > Sept > > >> 17. > > >> > >
> >>
> > > Interested parties, please plan to attend, if possible :) > > >> > > >>
> > > Cheers, > > >> Dmitri. > > >> > > >> On Wed, Sep 9, 2026 at 1:48 PM
> > Dmitri
> > > Bourlatchkov <di...@apache.org> > > >> wrote: > > >> > > >>> Hi JB, >
> >
> > > >>> > > >>> I've added an agenda item to the next community sync call
> for
> > > this > > (Sept > > >>> 17, 2026). > > >>> > > >>> Re: key points: did
> you
> > > mean a real GH discussion or dev email (this > > >>> thread)? Just
> double
> > > checking :) I'm fine with either approach. > > >>> > > >>> Cheers, > >
> > >>>
> > > Dmitri. > > >>> > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste
> > > Onofré < > jb@nanthrax.net> > > >>> wrote: > > >>> > > >>>> Hi
> Dmitri, >
> > > > >>>> > > >>>> I agree on the need to align on these points, though
> I'm
> > > not > entirely > > >>>> sure a dedicated meeting is necessary. Let's
> > start
> > > by using some > time > > >>>> during the next community meeting to
> > discuss
> > > it. > > >>>> > > >>>> In the meantime, what if we outline the key
> points
> > in
> > > a GitHub > > >>>> Discussion first? We can then use that as a reference
> > > during the > > >>>> meeting. > > >>>> > > >>>> Regards, > > >>>> JB > >
> > > >>>> > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov < >
> > > dimas@apache.org > > > > > >>>> wrote: > > >>>> > > > >>>> > Hi All,
> > >
> > > >>>> > > > >>>> > My impression from this thread is that it might be
> time
> > > for a call > > to > > >>>> > discuss all the related issues and try to
> > > arrive at a shared > > >>>> implementation > > >>>> > plan. > > >>>> >
> >
> > >
> > > >>>> > I know meetings are not ideal, but in this case it might be > >
> > >>>>
> > > beneficial as a > > >>>> > means for achieving a common understanding
> of
> > > the set of problems > > and > > >>>> > priorities related to this
> > thread. A
> > > dedicated metrics meeting > > worked > > >>>> well > > >>>> > from my
> > POV.
> > > > > >>>> > > > >>>> > Allocating a time slot in the next community
> > meeting
> > > might be an > > >>>> option, > > >>>> > although I think we might need
> > the
> > > full hour in this case. > > >>>> > > > >>>> > WDYT? > > >>>> > > >
> >>>> >
> > > Thanks, > > >>>> > Dmitri. > > >>>> > > > >>>> > On Mon, Aug 17, 2026
> at
> > > 6:39 PM Prithvi S < > > >>>> prithvisivasankar@gmail.com> > > >>>> >
> > > wrote: > > >>>> > > > >>>> > > Hi all, > > >>>> > > > > >>>> > > I
> > rewrote
> > > https://github.com/apache/polaris/pull/5035, > following > > >>>> the
> >
> > >
> > > >>>> > > discussion to focus only on the SPI foundation. > > >>>> > >
> > >
> > > >>>> > > What is in the PR now: > > >>>> > > > > >>>> > > - A written
> > > manager-level consistency contract in > > >>>> > >
> > > site/content/in-dev/unreleased/persistence-consistency.md. > > >>>> >
> > -
> > > CAS-aware EntityMutation / GrantMutation records. > > >>>> > > -
> > > MetaStoreChangeSet and BasePersistence#commitChangeSet, > with > > >>>>
> > > backends > > >>>> > > opting in via supportsAtomicMixedCommit(). > >
> > >>>> >
> > > > - commitChangeSet implementations for the in-memory TreeMap > > >>>>
> > > backend and > > >>>> > > JDBC. > > >>>> > > -
> > > AtomicOperationMetaStoreManager and > > >>>>
> > > TransactionalMetaStoreManagerImpl > > >>>> > > grant/revoke paths now
> > build
> > > one change set per operation, > > >>>> falling back > > >>>> > > to > >
> > > >>>> > > individual operations when the backend does not support mixed
> >
> > >
> > > >>>> commits. > > >>>> > > - Tests for the change-set API and TreeMap
> > > atomic commits. > > >>>> > > > > >>>> > > I intentionally left out: > >
> > > >>>> > > > > >>>> > > - Entity deletes in MetaStoreChangeSet. > > >>>>
> >
> > >
> > > - Refactoring createCatalog, dropEntity, and renameEntity to > > use >
> >
> > > >>>> change > > >>>> > > sets. > > >>>> > > - NoSQL support for
> > > commitChangeSet. > > >>>> > > > > >>>> > > I am treating this as Phase
> 1
> > so
> > > the contract and the SPI > > primitive > > >>>> can be > > >>>> > >
> > > reviewed before the larger operation migrations land. > > >>>> > > > >
> > >>>>
> > > > > Could you please take a look? In particular, I would like to > know
> > > >
> > > >>>> whether > > >>>> > > the contract captures the consensus so far,
> or
> > if
> > > it needs to > > >>>> address retry > > >>>> > > / external-work
> > > coordination before we merge this foundation. > > >>>> > > > > >>>> > >
> > > Thanks, > > >>>> > > Prithvi S > > >>>> > > > > >>>> > > On Mon, Aug
> 17,
> > > 2026 at 7:40 PM Dmitri Bourlatchkov < > > >>>> dimas@apache.org> > >
> > >>>>
> > > > > wrote: > > >>>> > > > > >>>> > > > Hi Robert, > > >>>> > > > > >
> > >>>> >
> > > > > I agree that the client-visible operation (e.g. REST API > >
> > request) >
> > > > >>>> > > consists > > >>>> > > > of many distinct phases. The
> database
> > /
> > > persistence changes > are > > >>>> just one > > >>>> > > of > > >>>> >
> >
> > >
> > > these phases. STS (as an example) is another. My point about > > >>>>
> > > transactions > > >>>> > > > in JDBC Persistence applies to the former
> > > (database changes) > > >>>> phase - one > > >>>> > > > retry attempt
> > there
> > > should ideally involve exactly one > > >>>> transaction. > > >>>> > > >
> > > >
> > > >>>> > > > We certainly need logic inside Polaris Servers to to >
> > > coordinate > > >>>> requests > > >>>> > > to > > >>>> > > > various
> > > external systems depending on outcomes from previous > > >>>> request
> > >
> > > >>>> > > > processing phases. > > >>>> > > > > > >>>> > > > I also
> agree
> > > that validation based on database state should be > > >>>> repeated > >
> > > >>>> > > > from scratch if we retry the database changes. > > >>>> > >
> >
> > >
> > > > >>>> > > > However, internal RBAC authorization naturally has to
> > happen >
> > > > >>>> inside the > > >>>> > > > database update phase
> > (read-check-write).
> > > IIRC, that is part > of > > >>>> the > > >>>> > > > implicit
> consistency
> > > guarantees, which were discussed during > > early > > >>>> > > project
> >
> > >
> > > >>>> > > > months (but never got written down in full clarity, > > >>>>
> > > unfortunately). If > > >>>> > > the > > >>>> > > > Authorization check
> is
> > > inside that phase for internal RBAC, I > > >>>> suppose it > > >>>> >
> > >
> > > will be there for other Authorizers too and that can involve > > >>>>
> > > external > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly
> > > expect > > >>>> authorization to be > > >>>> > > > efficient, but I'm
> not
> > > sure how quick it can be in practice. > > >>>> There's > > >>>> > > >
> > > certainly potential for delays in some (perhaps infrequent) > > cases.
> >
> > >
> > > >>>> > > > > > >>>> > > > Cheers, > > >>>> > > > Dmitri. > > >>>> > > >
> > > >
> > > >>>> > > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp
> <
> > > snazy@snazy.de> > > >>>> wrote: > > >>>> > > > > > >>>> > > > > Hi
> > > Dmitri, > > >>>> > > > > > > >>>> > > > > I agree that one logical
> > > operation needs a consistent view > of > > >>>> the whole > > >>>> > >
> >
> > >
> > > backend state. > > >>>> > > > > What worries me is treating one long
> JDBC
> > > transaction as the > > >>>> boundary > > >>>> > > of > > >>>> > > > >
> > that
> > > operation and then retrying the complete request if the > > >>>> commit
> > > >
> > > >>>> > > > fails. > > >>>> > > > > > > >>>> > > > > The request can
> also
> > > write metadata to object storage or > call > > >>>> STS. > > >>>> > >
> > >
> > > Each system can leave us with an outcome whose certainty we > > >>>>
> > cannot
> > > > > >>>> > > know. > > >>>> > > > > The database may have committed
> > before
> > > the connection was > > lost, > > >>>> an > > >>>> > > > > object-store
> > > write may have succeeded before a timeout, or > STS > > >>>> may have
> > >
> > > >>>> > > > > issued credentials before its response was lost. > > >>>>
> >
> > >
> > > > > A database rollback cannot undo any of those effects, and > > >>>>
> > > retrying the > > >>>> > > > > whole request may repeat them. > > >>>> >
> > > >
> > > > > > >>>> > > > > So I think the contract has to separate the overall
> >
> > > operation > > >>>> from each > > >>>> > > > > attempt to commit it. > >
> > > >>>> > > > > Each database or backend commit attempt should stay short.
> > > >
> > > >>>> > > > > When we retry, validation and authorization need to use
> the
> > >
> > > > >>>> state for > > >>>> > > > that > > >>>> > > > > new attempt, and
> we
> > > need clear rules for which external work > > >>>> can be > > >>>> > > >
> > > reused, > > >>>> > > > > repeated, cleaned up, or reconciled after an
> > > uncertain > > outcome. > > >>>> > > > > > > >>>> > > > > Some
> contextual
> > > state could be useful for carrying that > state. > > >>>> > > > > But
> the
> > > JDBC connection and transaction should be an > > >>>> implementation >
> >
> > > >>>> > > > detail > > >>>> > > > > inside it, not the boundary of the
> > > complete operation. > > >>>> > > > > > > >>>> > > > > Cheers, > > >>>>
> >
> > >
> > > > > Robert > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > On Mon,
> Aug
> > > 10, 2026 at 11:04 PM Dmitri Bourlatchkov < > > >>>> dimas@apache.org
> > >
> > > >>>> > > > > > >>>> > > > > wrote: > > >>>> > > > > > > >>>> > > > > >
> Hi
> > > All, > > >>>> > > > > > > > >>>> > > > > > The idea of a
> backend-agnostic
> > > change-set primitive sounds > > >>>> good. > > >>>> > > > > > > > >>>>
> >
> > >
> > > > > > However, I am not sure it is sufficient for all the use > > cases
> > > >
> > > >>>> we > > >>>> > > > touched > > >>>> > > > > > here. More
> > specifically,
> > > the state read by validation code > > is > > >>>> not > > >>>> > > > >
> >
> > > necessarily reflected in the change set: > > >>>> > > > > > > > >>>> >
> >
> > >
> > > > > * Unchanged entities might be considered by validation, > but > >
> > >>>>
> > > changed > > >>>> > > in > > >>>> > > > a > > >>>> > > > > > parallel
> > > request. > > >>>> > > > > > * Even for changed entities the state read
> by
> > > validation > may > > >>>> differ > > >>>> > > > from > > >>>> > > > > >
> > the
> > > state being altered. Some validation code talks to the > > >>>>
> > MetaStore >
> > > > >>>> > > > > > directly, outside of the data produced by the
> Resolver.
> > >
> > > > >>>> > > > > > > > >>>> > > > > > I do not think Polaris offers any
> > > explicit mechanisms > (ATM) > > >>>> to ensure > > >>>> > > > > >
> > > consistency between these reads and subsequent writes. > > >>>> > > > >
> > > >
> > > > >>>> > > > > > As far as JDBC goes, running an overarching
> Transaction
> > >
> > > > >>>> across all > > >>>> > > > reads > > >>>> > > > > > and writes in
> > the
> > > same request, at the SERIALIZABLE > > isolation > > >>>> level > > >>>>
> > > >
> > > in > > >>>> > > > > the > > >>>> > > > > > backing RDBMS could solve
> the
> > > problem, I think. > > >>>> > > > > > > > >>>> > > > > > Indeed, such a
> > > transaction might be long to accommodate > > calls > > >>>> made by > >
> > > >>>> > > > > > Polaris to external storage, etc. However, is that a > >
> > > problem? > > >>>> I think > > >>>> > > > the > > >>>> > > > > >
> > alternative
> > > is for the JDBC Persistence impl. to > "manually" > > >>>> track all >
> >
> > > >>>> > > > > reads > > >>>> > > > > > under the same request and redo
> > them
> > > in the small > > transaction > > >>>> that > > >>>> > > > > persists >
> >
> > > >>>> > > > > > the writes. This will also add RDBMS overhead and
> require
> > >
> > > > >>>> complex > > >>>> > > code > > >>>> > > > in > > >>>> > > > > >
> > > Polaris to handle the data properly. > > >>>> > > > > > > > >>>> > > >
> >
> > >
> > > Tracking the request-wide transaction can be done only for > > >>>>
> JDBC
> > >
> > > > >>>> > > without > > >>>> > > > > > leaking "transaction" concepts to
> > the
> > > NoSQL Persistence, I > > >>>> think. > > >>>> > > NoSQL > > >>>> > > >
> >
> > >
> > > will use other mechanisms to ensure read/write > consistency. > > >>>>
> >
> > >
> > > > > > > > >>>> > > > > > Connecting to Robert's email (a parallel
> branch
> > in
> > > this > > >>>> discussion), > > >>>> > > > I'd > > >>>> > > > > > like
> to
> > > propose this approach: > > >>>> > > > > > > > >>>> > > > > > * Each
> > request
> > > establishes a "Data Context" > > >>>> > > > > > - In JDBC the Data
> > Context
> > > corresponds to a JDBC > > Connection > > >>>> + Tx > > >>>> > > > > > -
> > In
> > > NoSQL the Data Context tracks one or more reference > > >>>> hashes > >
> > > >>>> > > > > > * All Persistence access in the same request goes
> through
> > >
> > > > the > > >>>> same > > >>>> > > Data > > >>>> > > > > > Context > >
> > >>>> >
> > > > > > > * All Persistence changes are committed once at the end of > >
> > the
> > > > > >>>> > > request > > >>>> > > > > > - Not all changes have to be
> > > globally atomic. For > example, > > >>>> changes > > >>>> > > in > >
> > >>>> >
> > > > > > > Catalogs A and B do not have to be atomic with respect to > >
> > >>>>
> > > each other. > > >>>> > > > We > > >>>> > > > > > can go deeper into
> this
> > > later. This is relevant to NoSQL. > > >>>> > > > > > - Transactional
> > > backends like JDBC can, of course, > choose > > >>>> to make > > >>>>
> > >
> > > all > > >>>> > > > > > changes globally atomic. > > >>>> > > > > > * A
> > > commit can fail in two main ways: > > >>>> > > > > > - A retriable
> > failure
> > > like an optimistic lock error or > Tx > > >>>> > > > > serializability
> >
> > >
> > > >>>> > > > > > error > > >>>> > > > > > - A non-triable logical error
> > (e.g.
> > > entity not found) > > >>>> > > > > > * On a retriable error the whole
> > > request is re-attempted > (as > > >>>> if > > >>>> > > > > resubmitted
> >
> > >
> > > >>>> > > > > > by a client) a few times (configurable timeout). > >
> >>>>
> > >
> > > > > > > > > >>>> > > > > > WDYT? > > >>>> > > > > > > > >>>> > > > > >
> > > Cheers, > > >>>> > > > > > Dmitri. > > >>>> > > > > > > > >>>> > > > >
> >
> > On
> > > Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré < > > >>>> > >
> > > jb@nanthrax.net > > >>>> > > > > > > >>>> > > > > > wrote: > > >>>> >
> >
> > >
> > > > > > > >>>> > > > > > > Hi all > > >>>> > > > > > > > > >>>> > > > > >
> > > I
> > > think Robert has a good point. > > >>>> > > > > > > > > >>>> > > > > >
> >
> > If
> > > the atomicity guarantee lives only in > BasePersistence, > > >>>> then
> > the
> > > > > >>>> > > > > > > manager contract can't tell a caller whether the
> > state
> > > > it > > >>>> read for > > >>>> > > > > > >
> > > validation/authorization/credential-vending is the same > > >>>> state
> > that
> > > > > >>>> > > > > > > eventually commits. That's the actual bug class
> > behind
> > > > the > > >>>> JDBC > > >>>> > > > > > > symptoms (not any single
> > operation
> > > being non-atomic, but > > the > > >>>> > > contract > > >>>> > > > > >
> >
> > > being silent about it. Every operation-specific fix > (like > > >>>>
> > #4939,
> > > > > >>>> > > > #5035 > > >>>> > > > > > > or #5095) re-answers this
> > > question locally and it keeps > > >>>> recurring. > > >>>> > > So > >
> > >>>>
> > > > > > > > > the deliverable should start with a written consistency > >
> > > >>>> contract at > > >>>> > > > > > > the manager level. > > >>>> > > >
> > > >
> > > > > > >>>> > > > > > > I'm not sure migrating everyone to > > >>>> > >
> > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > is a good idea.
> It
> > > would tie the logical change set to a > > DB > > >>>> > > > > > >
> > > transaction spanning the REST request. It means: > > >>>> > > > > > > -
> > it
> > > holds a durable transaction open across slow > external > > >>>> work
> > >
> > > >>>> > > > > > > (credential vending, OPA, Ranger, ...) > > >>>> > > >
> >
> > >
> > > > - it doesn't map to NoSQL > > >>>> > > > > > > - It wraps single-row
> > > updates in runWiithinTransaction, > > >>>> which is a > > >>>> > > > >
> >
> > > overhead > > >>>> > > > > > > > > >>>> > > > > > > So, I think the
> > > transactional manager isn't a portable > > >>>> target. It's > > >>>> >
> > > >
> > > > > > "only" one backend's strategy. > > >>>> > > > > > > > > >>>> > >
> >
> > >
> > > > > I think Privthi's approach is right. A backend-agnostic > > >>>>
> > > change-set > > >>>> > > > > > > primitive with a documented fallback is
> > the
> > > correct > shape. > > >>>> There is > > >>>> > > > > > > one caveat:
> > without
> > > carrying the original entitiy (not > > just > > >>>> the new > > >>>> >
> > > >
> > > > > > one) the change set can't express optimistic > concurrency. > >
> > >>>>
> > > > > > > > > > > >>>> > > > > > > I propose the following multi-steps
> > > approach: > > >>>> > > > > > > 1. We write the manager-level
> consistency
> > > contract as > > >>>> Robert asked. > > >>>> > > > It > > >>>> > > > >
> > >
> > > should include the explicit statement that a logical > > change > >
> >>>>
> > > set is > > >>>> > > > not > > >>>> > > > > > > a request-scope DB
> > > transaction. > > >>>> > > > > > > 2. We make Compare And Swap
> > > (optimistic-concurrency > > pattern) > > >>>> > > baseline > > >>>> > >
> > > >
> > > > > first-class in EntityMutation before merging the SPI > > >>>> > > >
> > > >
> > > > 3. Then we refactor > createCatalog/dropEntity/renameEntity. > > >>>>
> > > >
> > > > > > > > > >>>> > > > > > > Thoughts? > > >>>> > > > > > > > > >>>> >
> >
> > >
> > > > > > Regards > > >>>> > > > > > > JB > > >>>> > > > > > > > > >>>> > >
> > > >
> > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp < > > >>>>
> > snazy@snazy.de>
> > > > > >>>> > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > Hi
> > all, >
> > > > >>>> > > > > > > > > > >>>> > > > > > > > Yufei's clarification about
> > > where the atomicity > > guarantee > > >>>> is > > >>>> > > > defined >
> >
> > > >>>> > > > > > > seems > > >>>> > > > > > > > important. > > >>>> > > >
> > > >
> > > > > If it is a property of the lower-level BasePersistence > > >>>>
> > > contract > > >>>> > > > > rather > > >>>> > > > > > > than > > >>>> > >
> > > >
> > > > > > the general PolarisMetaStoreManager contract, the > > general > >
> > > >>>> > > contract > > >>>> > > > > does > > >>>> > > > > > > not > >
> > >>>> >
> > > > > > > > > tell callers whether the state used for validation, > >
> >>>>
> > >
> > > > authorization, > > >>>> > > > or > > >>>> > > > > > > > credential
> > > vending is consistent with the change that > > >>>> eventually > > >>>>
> > > >
> > > > > > commits. > > >>>> > > > > > > > > > >>>> > > > > > > > The
> current
> > > PRs suggest that operation-specific > > >>>> multi-object > > >>>> > >
> >
> > > methods > > >>>> > > > > > can > > >>>> > > > > > > > fix individual
> > cases,
> > > while leaving the same contract > > >>>> question to > > >>>> > > > >
> > recur
> > > > > >>>> > > > > > > for > > >>>> > > > > > > > each new case. > > >>>>
> > > >
> > > > > > > > > > >>>> > > > > > > > I would also avoid defining a logical
> > > change set as a > > >>>> database > > >>>> > > > > > > transaction > >
> > >>>>
> > > > > > > > > > around an entire REST request. > > >>>> > > > > > > >
> That
> > > would tie the contract to the backend and > database, > > >>>> and > >
> > >>>>
> > > > > could > > >>>> > > > > keep > > >>>> > > > > > > the > > >>>> > > >
> > > >
> > > > > durable attempt open across slow external work. > > >>>> > > > > >
> >
> > >
> > > > > >>>> > > > > > > > So is the choice really between the two current
> >
> > > manager > > >>>> > > > > > implementations, > > >>>> > > > > > > or > >
> > > >>>> > > > > > > > do we first need to revisit the boundary and >
> > > guarantees > > >>>> exposed > > >>>> > > to > > >>>> > > > > > their >
> >
> > > >>>> > > > > > > > callers? > > >>>> > > > > > > > > > >>>> > > > > >
> > >
> > > Cheers, > > >>>> > > > > > > > Robert > > >>>> > > > > > > > > > >>>> >
> > > >
> > > > > > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri
> > > Bourlatchkov < > > >>>> > > > > dimas@apache.org > > >>>> > > > > > >
> >
> > >
> > > >>>> > > > > > > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > >
> > >
> > > Hi Yufei, > > >>>> > > > > > > > > > > >>>> > > > > > > > > Thanks for
> > the
> > > link. I stand corrected. The "one > > atomic > > >>>> change > > >>>> >
> > > >
> > > per > > >>>> > > > > > > method" > > >>>> > > > > > > > > contract as
> > > defined in javadoc does apply only to > > >>>> > > BasePersistence > >
> > >>>>
> > > > > > > > > > > and AtomicOperationMetaStoreManager (which delegates >
> >
> > to
> > > > > >>>> > > > > > > BasePersistence). > > >>>> > > > > > > > > > >
> >>>>
> > >
> > > > > > > > > > Note that all non-test Persistence implementations > in
> > >
> > > >>>> the > > >>>> > > Polaris > > >>>> > > > > > > codebase > > >>>> >
> >
> > >
> > > > > > > > extend those classes (which is probably why I was > > >>>>
> > > confused > > >>>> > > about > > >>>> > > > > > > atomicity > > >>>> > >
> > > >
> > > > > > > expectations). > > >>>> > > > > > > > > > > >>>> > > > > > > >
> >
> > > However, this creates a gap in the Persistence SPI > > >>>> > >
> > > specification. > > >>>> > > > > If > > >>>> > > > > > > > > other
> > > PolarisMetaStoreManager implementations do not > > >>>> have to > >
> >>>>
> > >
> > > > > comply > > >>>> > > > > > > with > > >>>> > > > > > > > > this
> > > principle, it will create a conceptual > difficulty > > >>>> at call >
> >
> > > >>>> > > > > sites. > > >>>> > > > > > > How > > >>>> > > > > > > > >
> can
> > > PolarisMetaStoreManager callers reason about > > >>>> consistency > >
> > >>>>
> > > > > and > > >>>> > > > > > > durability > > >>>> > > > > > > > >
> > behaviours
> > > in general? > > >>>> > > > > > > > > > > >>>> > > > > > > > > I believe
> > we
> > > need to address that as part of this > > >>>> discussion. > > >>>> > >
> >
> > >
> > > > > > > > > >>>> > > > > > > > > Cheers, > > >>>> > > > > > > > >
> > Dmitri. >
> > > > >>>> > > > > > > > > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at
> > > 9:29 PM Yufei Gu < > > >>>> flyrain000@gmail.com > > >>>> > > > > >
> >>>>
> > > > > > > > wrote: > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >
> > The
> > > MetaStore SPI is currently defined with the > > >>>> idea that > > >>>>
> > > >
> > > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>>
> >
> > >
> > > > > > > > > > means one atomic change. > > >>>> > > > > > > > > > > >
> > >>>>
> > > > > > > > > > > > If the "MetaStore SPI" refers to the interface > >
> > >>>> >
> > > > > > > > PolarisMetaStoreManager, I > > >>>> > > > > > > > > > don't
> > think
> > > we've ever state each method to be > > >>>> atomic. We did > > >>>> > >
> > > >
> > > > clarify > > >>>> > > > > > > > > > atomicity[1] in the interface
> > > BasePersistence > > though. > > >>>> > > > > > > > > > > > >>>> > > > >
> > > >
> > > > > > > > >>>> > > > > > > > > > 1. > > >>>> > > > > > > > > > > > >>>>
> > > >
> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> >
> >
> > >
> > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > >
> > >
> >
> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > >
> >
> > >
> > > > > > > > >>>> > > > > > > > > > Yufei > > >>>> > > > > > > > > > > >
> > >>>>
> > > > > > > > > > > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at
> > > 11:27 AM Dmitri > > Bourlatchkov < > > >>>> > > > > > >
> dimas@apache.org
> > >
> > > > > >>>> > > > > > > > > > wrote: > > >>>> > > > > > > > > > > > >>>> >
> > > >
> > > > > > > > > > Hi Yufei, > > >>>> > > > > > > > > > > > > >>>> > > > > >
> > > >
> > > > > > I agree that JDBC transactions must be handled > > more > > >>>>
> >
> > >
> > > > explicitly. > > >>>> > > > > > > > > However, > > >>>> > > > > > > >
> >
> > >
> > > > I'm not sure that simply moving to > > >>>> > > > > > >
> > > TransactionalMetaStoreManagerImpl is > > >>>> > > > > > > > > > >
> > > sufficient. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > >
> The
> > > MetaStore SPI is currently defined with the > > >>>> idea that > > >>>>
> > > >
> > > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>>
> >
> > >
> > > > > > > > > > > means one atomic change [1]. The "transactional" > >
> >>>>
> > > MetaStore > > >>>> > > > > impl. > > >>>> > > > > > is > > >>>> > > > >
> > > >
> > > > > but a > > >>>> > > > > > > > > > > sub-case of that. It cannot
> alter
> > > the high-level > > >>>> contract. > > >>>> > > > > > > > > > > > > >>>>
> > > >
> > > > > > > > > > > We could add SPI methods having multiple object > >
> >>>>
> > > parameters > > >>>> > > to > > >>>> > > > > > > represent > > >>>> > >
> >
> > >
> > > > > > > > > grouped changes, but I am not sure it will be a > > >>>>
> > sound
> > > > > >>>> > > design. > > >>>> > > > > > This > > >>>> > > > > > > will
> >
> > >
> > > >>>> > > > > > > > > > > bloat the interface surfaces and require extra
> > > >
> > > >>>> impl. effort > > >>>> > > > for > > >>>> > > > > > > each > > >>>>
> > > >
> > > > > > > > > > > backend type. More importantly, adding multi-arg > >
> >>>>
> > > change > > >>>> > > > methods > > >>>> > > > > > > still > > >>>> > > >
> > > >
> > > > > > > won't > > >>>> > > > > > > > > > > address the problem of reads
> > > being consistent > with > > >>>> writes, > > >>>> > > > > > because > >
> > > >>>> > > > > > > each > > >>>> > > > > > > > > > > method call will
> still
> > > be independent regarding > > the > > >>>> data > > >>>> > > > stored >
> >
> > > >>>> > > > > > in > > >>>> > > > > > > the > > >>>> > > > > > > > > > >
> > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > I
> > tend
> > > to think we need to introduce a "change > > set" > > >>>> or > > >>>> >
> > > >
> > > "atomic > > >>>> > > > > > > batch" > > >>>> > > > > > > > > > >
> concept
> > to
> > > core Persistence and associate each > > REST > > >>>> API > > >>>> > >
> >
> > > request > > >>>> > > > > > > with > > >>>> > > > > > > > > one > > >>>>
> > > >
> > > > > > > > > > > such change set, which will be committed (or > > rolled
> > > >
> > > >>>> back) > > >>>> > > at > > >>>> > > > > the > > >>>> > > > > > >
> end
> > of
> > > > > >>>> > > > > > > > > > the > > >>>> > > > > > > > > > > request. I
> > > believe Ayush mentioned a similar > > >>>> concept in PR > > >>>> > > >
> > > 4939 > > >>>> > > > > > > [2]. In > > >>>> > > > > > > > > > > JDBC
> each
> > > change set will naturally be > associated > > >>>> with an > > >>>> >
> > >
> > > RDBMS > > >>>> > > > > > > > > > > transaction. In NoSQL persistence,
> > each
> > > atomic > > >>>> change set > > >>>> > > > will > > >>>> > > > > be > >
> > >>>>
> > > > > > > > > > > > > associated with one CAS operation on the > >
> > underlying
> > > > > >>>> > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > >
> >
> > >
> > > > > > [1] > > >>>> > > > > > > > > >>>>
> > > https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > >
> > >>>>
> > > > > > > > > > > > > > > >>>> > > > > > > > > > > [2] > > >>>> > > > > >
> > > >
> > > > >>>>
> > https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > Cheers, > >
> > >>>>
> > > > > > > > > > > > > Dmitri. > > >>>> > > > > > > > > > > > > >>>> > > >
> > > >
> > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < > > >>>> > > >
> > > flyrain000@gmail.com > > >>>> > > > > > > > >>>> > > > > > > wrote: >
> >
> > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > Thanks for
> > raising
> > > this, Dmitri. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > >
> > > >
> > > These are valid concerns, and they were > already > > >>>> > >
> > recognized >
> > > > >>>> > > > > when > > >>>> > > > > > > we > > >>>> > > > > > > > > >
> >
> > >
> > > introduced JDBC persistence to Polaris. At > that > > >>>> time, we > >
> > > >>>> > > > > chose > > >>>> > > > > > > to use > > >>>> > > > > > > > >
> > > >
> > > > AtomicOperationMetaStoreManager for the JDBC > due > > >>>> to the >
> >
> > > >>>> > > > > > > simplicity. I > > >>>> > > > > > > > > > > think > >
> > >>>>
> > > > > > > > > > > > > > most of the issues mentioned here can already >
> be
> > >
> > > > >>>> > > addressed > > >>>> > > > by > > >>>> > > > > > > > > > > >
> > > TransactionalMetaStoreManagerImpl. > > >>>> > > > > > > > > > > > > >
> > >>>>
> > > > > > > > > > > > > > For example, rename is already wrapped in a > >
> > >>>>
> > > transaction > > >>>> > > in > > >>>> > > > > > > > > > > >
> > > TransactionalMetaStoreManagerImpl [1]. > > Similarly, > > >>>> catalog
> >
> > >
> > > >>>> > > > > > > creation, > > >>>> > > > > > > > > > which > > >>>> >
> >
> > >
> > > > > > > > > > > involves reading and creating multiple > objects, > >
> > >>>>
> > > is also > > >>>> > > > > > executed > > >>>> > > > > > > > > > within a
> > > >
> > > >>>> > > > > > > > > > > > transaction [2]. > > >>>> > > > > > > > > >
> >
> > >
> > > > > >>>> > > > > > > > > > > > I see two possible directions: > > >>>>
> >
> > >
> > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > >
> >
> > >
> > > > > > > > > >>>> > > > > > > > > > > > Modify
> > > AtomicOperationMetaStoreManager > > >>>> together with > > >>>> > > the
> > > >
> > > >>>> > > > > > > > > persistence > > >>>> > > > > > > > > > > >
> backends
> > > (such as JDBC) to provide the > > required > > >>>> > > > > consistency
> > > >
> > > >>>> > > > > > > > > > guarantees > > >>>> > > > > > > > > > > > for >
> >
> > > >>>> > > > > > > > > > > > specific operations, similar to what > >
> >>>>
> > >
> > > > > > > > > > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > >
> >
> > >
> > > > > > does. > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > >
> >
> > >
> > > > > > >>>> > > > > > > > > > > > Migrate the persistence backends (such
> > as
> > > > > >>>> JDBC) to use > > >>>> > > > > > > > > > > >
> > > TransactionalMetaStoreManagerImpl directly. > > We > > >>>> may > >
> >>>>
> > >
> > > > have > > >>>> > > > to > > >>>> > > > > > > deal > > >>>> > > > > > >
> > > >
> > > with > > >>>> > > > > > > > > > > > transactional semantic mismatches
> > > across > > >>>> different > > >>>> > > > > > persistence > > >>>> > > >
> > > >
> > > > > > > > backends. > > >>>> > > > > > > > > > > > For example, we
> would
> > > likely avoid using > > JDBC's > > >>>> > > > > > > > > >
> > > `runWithinTransaction` > > >>>> > > > > > > > > > > > for > > >>>> > >
> >
> > >
> > > > > > > > > > single row updates, which adds additional > > >>>>
> overhead
> > > and > > >>>> > > > > > > complexity > > >>>> > > > > > > > > > > >
> > without
> > > > > >>>> > > > > > > > > > > > benefits. > > >>>> > > > > > > > > > > >
> > > >
> > > >>>> > > > > > > > > > > > References: > > >>>> > > > > > > > > > > >
> > >
> > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > > >>>> >
> > > >
> > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > >
> > > >
> > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > >
> >
> > >
> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > >
> > >>>> >
> > > > > > > >>>> > > > > >>>> > > >
> > >
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > > > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > > > > >
> > >>>> >
> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > >
> > > >
> > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>>
> >
> > >
> > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > >
> >
> > > >>>> > > > > > >>>> > > > > >>>> > > >
> > >
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > Yufei > >
> > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > >
> >
> > >
> > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri > > >>>>
> > Bourlatchkov
> > > < > > >>>> > > > > > > > > dimas@apache.org> > > >>>> > > > > > > > >
> >
> > >
> > > > wrote: > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > >
> >
> > Hi
> > > all, > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > >
> > > Ayush and Prithvi recently contributed a > > couple > > >>>> of > >
> >>>>
> > >
> > > > > > > interesting > > >>>> > > > > > > PRs: > > >>>> > > > > > > > >
> >
> > >
> > > > > [4939], [5035]. > > >>>> > > > > > > > > > > > > > > >>>> > > > > >
> > > >
> > > > > > > > It looks like people are starting to > encounter > > >>>> >
> > >
> > > consistency > > >>>> > > > > > > issues > > >>>> > > > > > > > > in > >
> > > >>>> > > > > > > > > > > > > JDBC persistence. > > >>>> > > > > > > > >
> > > >
> > > > > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight
> > into
> > > the > > >>>> underlying > > >>>> > > > > issues. > > >>>> > > > > > >
> > They
> > > > > >>>> > > > > > > > > > offer > > >>>> > > > > > > > > > > > >
> > > incremental fixes that can work. However, I > > >>>> believe it > >
> >>>>
> > >
> > > > is > > >>>> > > > > > time > > >>>> > > > > > > for > > >>>> > > > >
> >
> > >
> > > > > > the > > >>>> > > > > > > > > > > > > Polaris community to review
> > and
> > > improve this > > >>>> area of > > >>>> > > the > > >>>> > > > > > >
> > > codebase > > >>>> > > > > > > > > > > > > holistically. > > >>>> > > >
> >
> > >
> > > > > > > > > > > > >>>> > > > > > > > > > > > > By this, I mean finding
> a
> > > solution that can > be > > >>>> applied > > >>>> > > to > > >>>> > > >
> >
> > > all > > >>>> > > > > > > > > > > > > persistence backends (in-memory,
> > JDBC,
> > > > NoSQL) > > >>>> and > > >>>> > > > addresses > > >>>> > > > > > >
> > these >
> > > > >>>> > > > > > > > > > > > > aspects: > > >>>> > > > > > > > > > > >
> >
> > >
> > > > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent >
> > > changes > > >>>> where > > >>>> > > the > > >>>> > > > > > > service >
> >
> > > >>>> > > > > > > > > > reads > > >>>> > > > > > > > > > > > > and
> > validates
> > > current catalog state, then > > >>>> commits a > > >>>> > > > > change
> >
> > >
> > > >>>> > > > > > > (e.g. > > >>>> > > > > > > > > > > > > name clashes
> > during
> > > renames). > > >>>> > > > > > > > > > > > > * Supporting consistent but
> > > independent > > changes > > >>>> to RBAC > > >>>> > > > > > grants > >
> > >>>>
> > > > > > > > > and > > >>>> > > > > > > > > > > > > MetaStore entities.
> This
> > > independence is > > >>>> needed to > > >>>> > > > > support > > >>>> >
> >
> > >
> > > > > > > > > > > > external authorizers like OPA and Ranger. > > >>>> >
> >
> > >
> > > > > > > > > > > > * Supporting atomic changes across multiple > > >>>>
> > > similar > > >>>> > > > > entities. > > >>>> > > > > > > > > > > > > *
> > > Supporting authorization-based filtering > of > > >>>> list > > >>>> >
> >
> > >
> > > > operations > > >>>> > > > > > > (cf. > > >>>> > > > > > > > > > > > >
> > > [4831]). > > >>>> > > > > > > > > > > > > * Supporting
> credential-vending
> > > decisions > that > > >>>> are > > >>>> > > rooted > > >>>> > > > > in
> > >
> > > >>>> > > > > > > the > > >>>> > > > > > > > > > > > > exact state of
> the
> > > catalog. > > >>>> > > > > > > > > > > > > * Supporting server-side
> > retries
> > > for > transient > > >>>> > > > persistence > > >>>> > > > > > >
> failures
> > >
> > > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability
> failures).
> > >
> > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Please
> > > share your comments and ideas. > > >>>> > > > > > > > > > > > > > >
> >>>>
> > >
> > > > > > > > > > > > > > [4831] > > >>>>
> > > https://github.com/apache/polaris/pull/4831 > > >>>> > > > > > > > >
> > >
> > > > > > > >>>> > > > > > > > > > > > > [4939] > > >>>>
> > > https://github.com/apache/polaris/pull/4939 > > >>>> > > > > > > > >
> > >
> > > > > > > >>>> > > > > > > > > > > > > [5035] > > >>>>
> > > https://github.com/apache/polaris/pull/5035 > > >>>> > > > > > > > >
> > >
> > > > > > > >>>> > > > > > > > > > > > > Thanks, > > >>>> > > > > > > > > >
> > > >
> > > > Dmitri > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > >
> >
> > >
> > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > >
> >
> > >
> > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> >
> > > >
> > > > > >>>> > > > > >>>> > > >>> > > >
> > >
> > >
> >
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi EJ,
Thanks for sharing the PoC!
I'll try to explore that code, but cannot promise a quick turnaround :)
Reviewing this may take me a couple of weeks (due to other priorities).
Cheers,
Dmitri.
On Mon, Oct 5, 2026 at 4:47 PM EJ Wang <ej...@gmail.com>
wrote:
> One detail I omitted: the PoC also includes FoundationDB and Spanner
> adapters alongside JDBC. They reuse the same business-aware Manager. The
> adapters implement generic storage operations without table, catalog or
> grant rules.
>
> This exercises both goals: addressing consistency problems such as
> concurrent creation at overlapping table locations, and reusing that logic
> across databases. The concurrency tests have run on FoundationDB,
> PostgreSQL and CockroachDB. Spanner coverage currently uses the emulator,
> with three concurrency tests skipped.
>
> These are targeted feasibility checks. Production validation and
> integration with Polaris’s existing NoSQL backend remain follow-up work.
>
> -ej
>
> On Mon, Oct 5, 2026 at 1:36 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > Hi Dmitri, all,
> >
> > Two requests can both check for overlapping table locations, see none,
> and
> > then create conflicting tables. We need the checks and the resulting
> writes
> > to be protected together.
> >
> > My proposal is to implement these rules once in shared Polaris code (the
> > Manager). This business-aware layer checks them in the transaction that
> > creates the table. Database adapters would implement a business-agnostic
> > interface (Primitives) for generic reads, writes and commit. They would
> not
> > implement table, catalog or grant rules, so adding a database reuses the
> > same Polaris behavior.
> >
> > This PoC assumes the database itself can protect the relevant reads and
> > writes in one serializable transaction. Calls to object storage or
> > credential services happen outside it.
> >
> > The Java PoC
> > <
> https://github.com/flyingImer/polaris/tree/prototype/durable-java-primitives/persistence/primitives
> >
> > reuses the existing transactional Manager and tests selected races,
> > authorization retries and uncertain commits. It uses a temporary data
> > layout and reads more data than we'd want in production. Existing NoSQL
> > integration, production tuning and migration remain follow-up work.
> >
> > I'm not proposing an additional request-wide Persistence Session yet.
> > Existing business-aware request code still owns authorization, external
> > work and retries.
> >
> > The PoC allows an already-authorized metadata change to finish. A fresh
> > request retry checks authorization again. An uncertain commit stops
> > automatic retries and retains files that may already be referenced. These
> > policies still need agreement.
> >
> > Would this separation be a useful starting point?
> >
> > -ej
> >
> >
> > On Tue, 22 Sep 2026 19:08:49 -0400, Dmitri Bourlatchkov <
> dimas@apache.org>
> > wrote:
> >
> > Hi EJ, > 1. Which backend assumptions should we design for? [...] I know
> I
> > used the term "transaction" a lot during the call, but the real challenge
> > is about establishing an SPI with clear contracts that could be
> implemented
> > by JDBC and NoSQL persistence using the database features natural to each
> > model. I am fairly confident that a JDBC impl. is possible with
> > Serializable transaction. We should be able to solve all the problems
> > raised in this thread. However, that will make in-memory caching
> impossible
> > because the database needs to see all the reads (and writes) in a Tx to
> be
> > able to ensure serializable guarantees. If a cache prevents a read from
> > hitting the database, that read (e.g. storage config) will not be a
> factor
> > in serializability. This connects to my earlier question about the
> > in-memory cache and JDBC (title: Usefulness of InMemoryEntityCache) Are
> > different approaches possible with JDBC? Probably. What is involved? This
> > depends on the SPI contract, I think. > 2. How do we protect the data
> used
> > by validation? [...] The point mentioned by Robert previously IIRC, is
> that
> > we need to ground all runtime decisions to a particular state of the
> > catalog. If at commit time the state of the catalog in the database
> shifts
> > from what was used during validation, the request has to restart from
> > scratch. Current JDBC Persistence does this only for relationships
> between
> > entities and grants and between versions of the same entity.
> Relationships
> > between table locations and storage config are not actualized for the
> > purpose of state consistency tracking. From my POV we need to begin by
> > formulating this Persistence state in the Persistence SPI terms. Then,
> > think how we can implement that for JDBC. NoSQL Persistence already has a
> > per-catalog state (not granular), so wiring the new SPI to NoSQL should
> not
> > be too hard, I hope. The state in the SPI does not have to be global to
> the
> > catalog (as in NoSQL), but it needs to be able to express the idea of
> which
> > elements were engaged during request processing so serve as input to
> > Persistence impl. for cosistency guarantees. If Persistence caches data
> in
> > memory (for example), it will have to do so in a way that does not
> violate
> > consistency expectations. From my POV we could begin by defining a
> > "Persistence Session" concept. Each request is 1:1 with a session. All
> > reads and writes go through the session. The next step would be the JDBC
> > impl., which probably needs some brainstorming... but I wonder if we
> could
> > reach a consensus at a high level first. WDYT? Thanks, Dmitri. On Tue,
> Sep
> > 22, 2026 at 5:55 PM EJ Wang <ej...@gmail.com> wrote: >
> > Thanks for the recap, Dmitri. I'd like to follow up on the three points
> I >
> > raised during the sync. We didn't have much time to discuss them, so it >
> > would help to confirm these before choosing an SPI design. > > 1. Which
> > backend assumptions should we design for? > > Within one Polaris server
> > instance, do we assume entities, grants and > secrets all use the same
> > database backend, or must we support mixing > backends across them? >
> Must
> > every supported persistence implementation provide transactional >
> > guarantees? Does the database itself need to provide transactions, or
> can >
> > Polaris build that capability on top of simpler database operations? > >
> > I'm happy to work with a homogeneous assumption if that's the intended >
> > scope. Existing NoSQL illustrates the second question: it can publish >
> > changes to multiple objects under one reference using a single
> conditional
> > > pointer update. > > 2. How do we protect the data used by validation?
> > >
> > Your storage-configuration and location-overlap examples make the
> problem >
> > concrete. Consider these races: > > A table creation passes validation
> > against the catalog's allowed locations, > then another request changes
> > that configuration before the table is > committed. > Two table-creation
> > requests each check for overlapping locations, both see > no overlap, and
> > then create tables at conflicting locations. > > Checking only the record
> > being written cannot cover these dependencies. The > first also depends
> on
> > the catalog configuration. The second depends on the > absence of another
> > overlapping table. > > Could the persistence layer protect these reads
> > together with the writes, > using backend-internal versions/tokens or
> > database isolation? Shared code > would still need to identify the
> required
> > reads and checks. Could we keep > the mechanism for protecting them
> inside
> > each backend? > > 3. Business rules are duplicated across backend
> > implementations > > Today, the TreeMap > < >
> >
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481
> > > > > and JDBC > < >
> >
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046
> > > > > implementations each load a secret, check its principal, apply the
> >
> > rotation/reset steps and write it back. > > Changing these business rules
> > therefore requires keeping multiple > implementations in sync. A fix
> > applied to one backend can be missed in > another, causing the same
> Polaris
> > operation to behave differently depending > on the database. > > My
> > proposal is to implement each business workflow once, against a common >
> > set of storage operations with defined consistency and failure
> semantics. >
> > The shared workflow should not need to know whether those operations are
> >
> > implemented through JDBC transactions, NoSQL reference CAS or another >
> > mechanism. Each backend would implement the same storage contract,
> keeping
> > > its execution details internal. A new database backend could then reuse
> > the > existing business workflows. > > Do these seem like reasonable
> > starting points? We can then evaluate the SPI > against the original
> > failure cases on both JDBC and existing NoSQL. > > -ej > > On Fri, Sep
> 18,
> > 2026 at 12:10 PM Dmitri Bourlatchkov <di...@apache.org> > wrote: > > >
> > Side note: I just came across [5541], which shows a case where a failed
> > >
> > request leaves persisted side effects. > > > > This can be seen as a
> coding
> > mistake, of course. However, this is about > > executing one request as a
> > single, atomic unit. This is something that > > probably needs
> foundational
> > support and coding patterns in Polaris code. > > > > Cheers, > > Dmitri.
> >
> > > [5541] https://github.com/apache/polaris/pull/5541 > > > > On Fri, Sep
> > 18, 2026 at 3:02 PM Dmitri Bourlatchkov <di...@apache.org> > > wrote: >
> >
> > > > > Hi All, > > > > > > Here's a recap of the Community Sync discussion
> > as I remember it. I'm > > sure > > > my recollection is not complete, so
> > please add your thoughts to this > > thread. > > > > > > * Polaris
> > traditionally relies on entity version number to ensure > > > consistency
> > of changes. > > > > > > This works well for internal RBAC grants, which
> > also have version > numbers > > > cross-referenced at the Persistence
> > layer. > > > > > > I do not think this works in more general cases like
> > validating > locations > > > wrt catalog-level storage configuration. >
> > >
> > > > > Another difficult case involves location overlaps between
> concurrent
> > > > table > > > creations. > > > > > > * We talked about how Persistence
> > calls related to RDBMS transactions > (in > > > the JDBC persistence
> case).
> > > > > > > > There were some concerns raised about whether we need to
> bind >
> > Persistence > > > API to transactions explicitly. > > > > > > I think it
> is
> > a valid concern. At the same time, the Persistence API > > > should be
> > clear about consistency guarantees across all backend > > >
> > implementations. I think this needs more attention now that the matter >
> of
> > > > > milti-entioty changes became prominent in [5035]. > > > > > > I
> > personally believe that RDBMS-based Persistence in Polaris has to use >
> > >
> > Serializable transaction isolation. Consequently, the core code needs >
> to
> > > > > provide mechanisms to allow the Persistence implementation to
> > properly > > > connect reads and writes from API requests to JDBC
> > transactions... > > whether > > > we use "transation" as an explicit
> terms
> > in the SPI or not. > > > > > > Note that currently, the same API request
> > performs reads and writes in > > > _separate_ RDBMS transactions (even
> > separate connections). > > > > > > * We did not discuss Persistence SPI
> > implications in the call, but > > > currently each Persistence call is
> > assumed to be one distinct and > atomic > > > change (please correct me
> if
> > I'm wrong). > > > > > > Consequently, supporting new use cases involves
> > introducing new SPI > > > methods with many parameters. This can be seen
> in
> > PR [5035]. This tends > > to > > > overcomplicate the SPI. > > > > > >
> > Ideally, I think the SPI should be simplified to deal with multi-entity
> > >
> > > and single-entity changes in a coherent manner to avoid any ambiguity
> > >
> > about > > > which method the caller should use. > > > > > > [5035]
> > https://github.com/apache/polaris/pull/5035 > > > > > > Cheers, > > >
> > Dmitri. > > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov
> <
> > dimas@apache.org > > > > > wrote: > > > > > >> Heads up: This discussion
> > is on the Community Sync call agenda for > Sept > > >> 17. > > >> > > >>
> > Interested parties, please plan to attend, if possible :) > > >> > > >>
> > Cheers, > > >> Dmitri. > > >> > > >> On Wed, Sep 9, 2026 at 1:48 PM
> Dmitri
> > Bourlatchkov <di...@apache.org> > > >> wrote: > > >> > > >>> Hi JB, > >
> > >>> > > >>> I've added an agenda item to the next community sync call for
> > this > > (Sept > > >>> 17, 2026). > > >>> > > >>> Re: key points: did you
> > mean a real GH discussion or dev email (this > > >>> thread)? Just double
> > checking :) I'm fine with either approach. > > >>> > > >>> Cheers, > >
> >>>
> > Dmitri. > > >>> > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste
> > Onofré < > jb@nanthrax.net> > > >>> wrote: > > >>> > > >>>> Hi Dmitri, >
> > > >>>> > > >>>> I agree on the need to align on these points, though I'm
> > not > entirely > > >>>> sure a dedicated meeting is necessary. Let's
> start
> > by using some > time > > >>>> during the next community meeting to
> discuss
> > it. > > >>>> > > >>>> In the meantime, what if we outline the key points
> in
> > a GitHub > > >>>> Discussion first? We can then use that as a reference
> > during the > > >>>> meeting. > > >>>> > > >>>> Regards, > > >>>> JB > >
> > >>>> > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov < >
> > dimas@apache.org > > > > > >>>> wrote: > > >>>> > > > >>>> > Hi All, > >
> > >>>> > > > >>>> > My impression from this thread is that it might be time
> > for a call > > to > > >>>> > discuss all the related issues and try to
> > arrive at a shared > > >>>> implementation > > >>>> > plan. > > >>>> > >
> >
> > >>>> > I know meetings are not ideal, but in this case it might be > >
> >>>>
> > beneficial as a > > >>>> > means for achieving a common understanding of
> > the set of problems > > and > > >>>> > priorities related to this
> thread. A
> > dedicated metrics meeting > > worked > > >>>> well > > >>>> > from my
> POV.
> > > > >>>> > > > >>>> > Allocating a time slot in the next community
> meeting
> > might be an > > >>>> option, > > >>>> > although I think we might need
> the
> > full hour in this case. > > >>>> > > > >>>> > WDYT? > > >>>> > > > >>>> >
> > Thanks, > > >>>> > Dmitri. > > >>>> > > > >>>> > On Mon, Aug 17, 2026 at
> > 6:39 PM Prithvi S < > > >>>> prithvisivasankar@gmail.com> > > >>>> >
> > wrote: > > >>>> > > > >>>> > > Hi all, > > >>>> > > > > >>>> > > I
> rewrote
> > https://github.com/apache/polaris/pull/5035, > following > > >>>> the >
> >
> > >>>> > > discussion to focus only on the SPI foundation. > > >>>> > > > >
> > >>>> > > What is in the PR now: > > >>>> > > > > >>>> > > - A written
> > manager-level consistency contract in > > >>>> > >
> > site/content/in-dev/unreleased/persistence-consistency.md. > > >>>> > > -
> > CAS-aware EntityMutation / GrantMutation records. > > >>>> > > -
> > MetaStoreChangeSet and BasePersistence#commitChangeSet, > with > > >>>>
> > backends > > >>>> > > opting in via supportsAtomicMixedCommit(). > >
> >>>> >
> > > - commitChangeSet implementations for the in-memory TreeMap > > >>>>
> > backend and > > >>>> > > JDBC. > > >>>> > > -
> > AtomicOperationMetaStoreManager and > > >>>>
> > TransactionalMetaStoreManagerImpl > > >>>> > > grant/revoke paths now
> build
> > one change set per operation, > > >>>> falling back > > >>>> > > to > >
> > >>>> > > individual operations when the backend does not support mixed >
> >
> > >>>> commits. > > >>>> > > - Tests for the change-set API and TreeMap
> > atomic commits. > > >>>> > > > > >>>> > > I intentionally left out: > >
> > >>>> > > > > >>>> > > - Entity deletes in MetaStoreChangeSet. > > >>>> >
> >
> > - Refactoring createCatalog, dropEntity, and renameEntity to > > use > >
> > >>>> change > > >>>> > > sets. > > >>>> > > - NoSQL support for
> > commitChangeSet. > > >>>> > > > > >>>> > > I am treating this as Phase 1
> so
> > the contract and the SPI > > primitive > > >>>> can be > > >>>> > >
> > reviewed before the larger operation migrations land. > > >>>> > > > >
> >>>>
> > > > Could you please take a look? In particular, I would like to > know
> > >
> > >>>> whether > > >>>> > > the contract captures the consensus so far, or
> if
> > it needs to > > >>>> address retry > > >>>> > > / external-work
> > coordination before we merge this foundation. > > >>>> > > > > >>>> > >
> > Thanks, > > >>>> > > Prithvi S > > >>>> > > > > >>>> > > On Mon, Aug 17,
> > 2026 at 7:40 PM Dmitri Bourlatchkov < > > >>>> dimas@apache.org> > >
> >>>>
> > > > wrote: > > >>>> > > > > >>>> > > > Hi Robert, > > >>>> > > > > >
> >>>> >
> > > > I agree that the client-visible operation (e.g. REST API > >
> request) >
> > > >>>> > > consists > > >>>> > > > of many distinct phases. The database
> /
> > persistence changes > are > > >>>> just one > > >>>> > > of > > >>>> > >
> >
> > these phases. STS (as an example) is another. My point about > > >>>>
> > transactions > > >>>> > > > in JDBC Persistence applies to the former
> > (database changes) > > >>>> phase - one > > >>>> > > > retry attempt
> there
> > should ideally involve exactly one > > >>>> transaction. > > >>>> > > >
> > >
> > >>>> > > > We certainly need logic inside Polaris Servers to to >
> > coordinate > > >>>> requests > > >>>> > > to > > >>>> > > > various
> > external systems depending on outcomes from previous > > >>>> request > >
> > >>>> > > > processing phases. > > >>>> > > > > > >>>> > > > I also agree
> > that validation based on database state should be > > >>>> repeated > >
> > >>>> > > > from scratch if we retry the database changes. > > >>>> > > >
> >
> > > >>>> > > > However, internal RBAC authorization naturally has to
> happen >
> > > >>>> inside the > > >>>> > > > database update phase
> (read-check-write).
> > IIRC, that is part > of > > >>>> the > > >>>> > > > implicit consistency
> > guarantees, which were discussed during > > early > > >>>> > > project >
> >
> > >>>> > > > months (but never got written down in full clarity, > > >>>>
> > unfortunately). If > > >>>> > > the > > >>>> > > > Authorization check is
> > inside that phase for internal RBAC, I > > >>>> suppose it > > >>>> > > >
> > will be there for other Authorizers too and that can involve > > >>>>
> > external > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly
> > expect > > >>>> authorization to be > > >>>> > > > efficient, but I'm not
> > sure how quick it can be in practice. > > >>>> There's > > >>>> > > >
> > certainly potential for delays in some (perhaps infrequent) > > cases. >
> >
> > >>>> > > > > > >>>> > > > Cheers, > > >>>> > > > Dmitri. > > >>>> > > >
> > >
> > >>>> > > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <
> > snazy@snazy.de> > > >>>> wrote: > > >>>> > > > > > >>>> > > > > Hi
> > Dmitri, > > >>>> > > > > > > >>>> > > > > I agree that one logical
> > operation needs a consistent view > of > > >>>> the whole > > >>>> > > >
> >
> > backend state. > > >>>> > > > > What worries me is treating one long JDBC
> > transaction as the > > >>>> boundary > > >>>> > > of > > >>>> > > > >
> that
> > operation and then retrying the complete request if the > > >>>> commit
> > >
> > >>>> > > > fails. > > >>>> > > > > > > >>>> > > > > The request can also
> > write metadata to object storage or > call > > >>>> STS. > > >>>> > > > >
> > Each system can leave us with an outcome whose certainty we > > >>>>
> cannot
> > > > >>>> > > know. > > >>>> > > > > The database may have committed
> before
> > the connection was > > lost, > > >>>> an > > >>>> > > > > object-store
> > write may have succeeded before a timeout, or > STS > > >>>> may have > >
> > >>>> > > > > issued credentials before its response was lost. > > >>>> >
> >
> > > > A database rollback cannot undo any of those effects, and > > >>>>
> > retrying the > > >>>> > > > > whole request may repeat them. > > >>>> >
> > >
> > > > > >>>> > > > > So I think the contract has to separate the overall >
> > operation > > >>>> from each > > >>>> > > > > attempt to commit it. > >
> > >>>> > > > > Each database or backend commit attempt should stay short.
> > >
> > >>>> > > > > When we retry, validation and authorization need to use the
> >
> > > >>>> state for > > >>>> > > > that > > >>>> > > > > new attempt, and we
> > need clear rules for which external work > > >>>> can be > > >>>> > > >
> > reused, > > >>>> > > > > repeated, cleaned up, or reconciled after an
> > uncertain > > outcome. > > >>>> > > > > > > >>>> > > > > Some contextual
> > state could be useful for carrying that > state. > > >>>> > > > > But the
> > JDBC connection and transaction should be an > > >>>> implementation > >
> > >>>> > > > detail > > >>>> > > > > inside it, not the boundary of the
> > complete operation. > > >>>> > > > > > > >>>> > > > > Cheers, > > >>>> >
> >
> > > > Robert > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > On Mon, Aug
> > 10, 2026 at 11:04 PM Dmitri Bourlatchkov < > > >>>> dimas@apache.org > >
> > >>>> > > > > > >>>> > > > > wrote: > > >>>> > > > > > > >>>> > > > > > Hi
> > All, > > >>>> > > > > > > > >>>> > > > > > The idea of a backend-agnostic
> > change-set primitive sounds > > >>>> good. > > >>>> > > > > > > > >>>> >
> >
> > > > > However, I am not sure it is sufficient for all the use > > cases
> > >
> > >>>> we > > >>>> > > > touched > > >>>> > > > > > here. More
> specifically,
> > the state read by validation code > > is > > >>>> not > > >>>> > > > > >
> > necessarily reflected in the change set: > > >>>> > > > > > > > >>>> > >
> >
> > > > * Unchanged entities might be considered by validation, > but > >
> >>>>
> > changed > > >>>> > > in > > >>>> > > > a > > >>>> > > > > > parallel
> > request. > > >>>> > > > > > * Even for changed entities the state read by
> > validation > may > > >>>> differ > > >>>> > > > from > > >>>> > > > > >
> the
> > state being altered. Some validation code talks to the > > >>>>
> MetaStore >
> > > >>>> > > > > > directly, outside of the data produced by the Resolver.
> >
> > > >>>> > > > > > > > >>>> > > > > > I do not think Polaris offers any
> > explicit mechanisms > (ATM) > > >>>> to ensure > > >>>> > > > > >
> > consistency between these reads and subsequent writes. > > >>>> > > > >
> > >
> > > >>>> > > > > > As far as JDBC goes, running an overarching Transaction
> >
> > > >>>> across all > > >>>> > > > reads > > >>>> > > > > > and writes in
> the
> > same request, at the SERIALIZABLE > > isolation > > >>>> level > > >>>>
> > >
> > in > > >>>> > > > > the > > >>>> > > > > > backing RDBMS could solve the
> > problem, I think. > > >>>> > > > > > > > >>>> > > > > > Indeed, such a
> > transaction might be long to accommodate > > calls > > >>>> made by > >
> > >>>> > > > > > Polaris to external storage, etc. However, is that a > >
> > problem? > > >>>> I think > > >>>> > > > the > > >>>> > > > > >
> alternative
> > is for the JDBC Persistence impl. to > "manually" > > >>>> track all > >
> > >>>> > > > > reads > > >>>> > > > > > under the same request and redo
> them
> > in the small > > transaction > > >>>> that > > >>>> > > > > persists > >
> > >>>> > > > > > the writes. This will also add RDBMS overhead and require
> >
> > > >>>> complex > > >>>> > > code > > >>>> > > > in > > >>>> > > > > >
> > Polaris to handle the data properly. > > >>>> > > > > > > > >>>> > > > >
> >
> > Tracking the request-wide transaction can be done only for > > >>>> JDBC
> >
> > > >>>> > > without > > >>>> > > > > > leaking "transaction" concepts to
> the
> > NoSQL Persistence, I > > >>>> think. > > >>>> > > NoSQL > > >>>> > > > >
> >
> > will use other mechanisms to ensure read/write > consistency. > > >>>> >
> >
> > > > > > > >>>> > > > > > Connecting to Robert's email (a parallel branch
> in
> > this > > >>>> discussion), > > >>>> > > > I'd > > >>>> > > > > > like to
> > propose this approach: > > >>>> > > > > > > > >>>> > > > > > * Each
> request
> > establishes a "Data Context" > > >>>> > > > > > - In JDBC the Data
> Context
> > corresponds to a JDBC > > Connection > > >>>> + Tx > > >>>> > > > > > -
> In
> > NoSQL the Data Context tracks one or more reference > > >>>> hashes > >
> > >>>> > > > > > * All Persistence access in the same request goes through
> >
> > > the > > >>>> same > > >>>> > > Data > > >>>> > > > > > Context > >
> >>>> >
> > > > > > * All Persistence changes are committed once at the end of > >
> the
> > > > >>>> > > request > > >>>> > > > > > - Not all changes have to be
> > globally atomic. For > example, > > >>>> changes > > >>>> > > in > >
> >>>> >
> > > > > > Catalogs A and B do not have to be atomic with respect to > >
> >>>>
> > each other. > > >>>> > > > We > > >>>> > > > > > can go deeper into this
> > later. This is relevant to NoSQL. > > >>>> > > > > > - Transactional
> > backends like JDBC can, of course, > choose > > >>>> to make > > >>>> > >
> > all > > >>>> > > > > > changes globally atomic. > > >>>> > > > > > * A
> > commit can fail in two main ways: > > >>>> > > > > > - A retriable
> failure
> > like an optimistic lock error or > Tx > > >>>> > > > > serializability >
> >
> > >>>> > > > > > error > > >>>> > > > > > - A non-triable logical error
> (e.g.
> > entity not found) > > >>>> > > > > > * On a retriable error the whole
> > request is re-attempted > (as > > >>>> if > > >>>> > > > > resubmitted >
> >
> > >>>> > > > > > by a client) a few times (configurable timeout). > > >>>>
> >
> > > > > > > > >>>> > > > > > WDYT? > > >>>> > > > > > > > >>>> > > > > >
> > Cheers, > > >>>> > > > > > Dmitri. > > >>>> > > > > > > > >>>> > > > > >
> On
> > Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré < > > >>>> > >
> > jb@nanthrax.net > > >>>> > > > > > > >>>> > > > > > wrote: > > >>>> > >
> >
> > > > > > >>>> > > > > > > Hi all > > >>>> > > > > > > > > >>>> > > > > >
> > I
> > think Robert has a good point. > > >>>> > > > > > > > > >>>> > > > > > >
> If
> > the atomicity guarantee lives only in > BasePersistence, > > >>>> then
> the
> > > > >>>> > > > > > > manager contract can't tell a caller whether the
> state
> > > it > > >>>> read for > > >>>> > > > > > >
> > validation/authorization/credential-vending is the same > > >>>> state
> that
> > > > >>>> > > > > > > eventually commits. That's the actual bug class
> behind
> > > the > > >>>> JDBC > > >>>> > > > > > > symptoms (not any single
> operation
> > being non-atomic, but > > the > > >>>> > > contract > > >>>> > > > > > >
> > being silent about it. Every operation-specific fix > (like > > >>>>
> #4939,
> > > > >>>> > > > #5035 > > >>>> > > > > > > or #5095) re-answers this
> > question locally and it keeps > > >>>> recurring. > > >>>> > > So > >
> >>>>
> > > > > > > > the deliverable should start with a written consistency > >
> > >>>> contract at > > >>>> > > > > > > the manager level. > > >>>> > > >
> > >
> > > > > >>>> > > > > > > I'm not sure migrating everyone to > > >>>> > >
> > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > is a good idea. It
> > would tie the logical change set to a > > DB > > >>>> > > > > > >
> > transaction spanning the REST request. It means: > > >>>> > > > > > > -
> it
> > holds a durable transaction open across slow > external > > >>>> work > >
> > >>>> > > > > > > (credential vending, OPA, Ranger, ...) > > >>>> > > > >
> >
> > > - it doesn't map to NoSQL > > >>>> > > > > > > - It wraps single-row
> > updates in runWiithinTransaction, > > >>>> which is a > > >>>> > > > > >
> > overhead > > >>>> > > > > > > > > >>>> > > > > > > So, I think the
> > transactional manager isn't a portable > > >>>> target. It's > > >>>> >
> > >
> > > > > "only" one backend's strategy. > > >>>> > > > > > > > > >>>> > > >
> >
> > > > I think Privthi's approach is right. A backend-agnostic > > >>>>
> > change-set > > >>>> > > > > > > primitive with a documented fallback is
> the
> > correct > shape. > > >>>> There is > > >>>> > > > > > > one caveat:
> without
> > carrying the original entitiy (not > > just > > >>>> the new > > >>>> >
> > >
> > > > > one) the change set can't express optimistic > concurrency. > >
> >>>>
> > > > > > > > > > >>>> > > > > > > I propose the following multi-steps
> > approach: > > >>>> > > > > > > 1. We write the manager-level consistency
> > contract as > > >>>> Robert asked. > > >>>> > > > It > > >>>> > > > > > >
> > should include the explicit statement that a logical > > change > > >>>>
> > set is > > >>>> > > > not > > >>>> > > > > > > a request-scope DB
> > transaction. > > >>>> > > > > > > 2. We make Compare And Swap
> > (optimistic-concurrency > > pattern) > > >>>> > > baseline > > >>>> > >
> > >
> > > > first-class in EntityMutation before merging the SPI > > >>>> > > >
> > >
> > > 3. Then we refactor > createCatalog/dropEntity/renameEntity. > > >>>>
> > >
> > > > > > > > >>>> > > > > > > Thoughts? > > >>>> > > > > > > > > >>>> > >
> >
> > > > > Regards > > >>>> > > > > > > JB > > >>>> > > > > > > > > >>>> > >
> > >
> > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp < > > >>>>
> snazy@snazy.de>
> > > > >>>> > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > Hi
> all, >
> > > >>>> > > > > > > > > > >>>> > > > > > > > Yufei's clarification about
> > where the atomicity > > guarantee > > >>>> is > > >>>> > > > defined > >
> > >>>> > > > > > > seems > > >>>> > > > > > > > important. > > >>>> > > >
> > >
> > > > If it is a property of the lower-level BasePersistence > > >>>>
> > contract > > >>>> > > > > rather > > >>>> > > > > > > than > > >>>> > >
> > >
> > > > > the general PolarisMetaStoreManager contract, the > > general > >
> > >>>> > > contract > > >>>> > > > > does > > >>>> > > > > > > not > >
> >>>> >
> > > > > > > > tell callers whether the state used for validation, > > >>>>
> >
> > > authorization, > > >>>> > > > or > > >>>> > > > > > > > credential
> > vending is consistent with the change that > > >>>> eventually > > >>>>
> > >
> > > > > commits. > > >>>> > > > > > > > > > >>>> > > > > > > > The current
> > PRs suggest that operation-specific > > >>>> multi-object > > >>>> > > >
> > methods > > >>>> > > > > > can > > >>>> > > > > > > > fix individual
> cases,
> > while leaving the same contract > > >>>> question to > > >>>> > > > >
> recur
> > > > >>>> > > > > > > for > > >>>> > > > > > > > each new case. > > >>>>
> > >
> > > > > > > > > >>>> > > > > > > > I would also avoid defining a logical
> > change set as a > > >>>> database > > >>>> > > > > > > transaction > >
> >>>>
> > > > > > > > > around an entire REST request. > > >>>> > > > > > > > That
> > would tie the contract to the backend and > database, > > >>>> and > >
> >>>>
> > > > could > > >>>> > > > > keep > > >>>> > > > > > > the > > >>>> > > >
> > >
> > > > durable attempt open across slow external work. > > >>>> > > > > > >
> >
> > > > >>>> > > > > > > > So is the choice really between the two current >
> > manager > > >>>> > > > > > implementations, > > >>>> > > > > > > or > >
> > >>>> > > > > > > > do we first need to revisit the boundary and >
> > guarantees > > >>>> exposed > > >>>> > > to > > >>>> > > > > > their > >
> > >>>> > > > > > > > callers? > > >>>> > > > > > > > > > >>>> > > > > > > >
> > Cheers, > > >>>> > > > > > > > Robert > > >>>> > > > > > > > > > >>>> >
> > >
> > > > > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri
> > Bourlatchkov < > > >>>> > > > > dimas@apache.org > > >>>> > > > > > > >
> >
> > >>>> > > > > > > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > >
> > Hi Yufei, > > >>>> > > > > > > > > > > >>>> > > > > > > > > Thanks for
> the
> > link. I stand corrected. The "one > > atomic > > >>>> change > > >>>> >
> > >
> > per > > >>>> > > > > > > method" > > >>>> > > > > > > > > contract as
> > defined in javadoc does apply only to > > >>>> > > BasePersistence > >
> >>>>
> > > > > > > > > > and AtomicOperationMetaStoreManager (which delegates > >
> to
> > > > >>>> > > > > > > BasePersistence). > > >>>> > > > > > > > > > > >>>>
> >
> > > > > > > > > Note that all non-test Persistence implementations > in > >
> > >>>> the > > >>>> > > Polaris > > >>>> > > > > > > codebase > > >>>> > >
> >
> > > > > > > extend those classes (which is probably why I was > > >>>>
> > confused > > >>>> > > about > > >>>> > > > > > > atomicity > > >>>> > >
> > >
> > > > > > expectations). > > >>>> > > > > > > > > > > >>>> > > > > > > > >
> > However, this creates a gap in the Persistence SPI > > >>>> > >
> > specification. > > >>>> > > > > If > > >>>> > > > > > > > > other
> > PolarisMetaStoreManager implementations do not > > >>>> have to > > >>>>
> >
> > > > comply > > >>>> > > > > > > with > > >>>> > > > > > > > > this
> > principle, it will create a conceptual > difficulty > > >>>> at call > >
> > >>>> > > > > sites. > > >>>> > > > > > > How > > >>>> > > > > > > > > can
> > PolarisMetaStoreManager callers reason about > > >>>> consistency > >
> >>>>
> > > > and > > >>>> > > > > > > durability > > >>>> > > > > > > > >
> behaviours
> > in general? > > >>>> > > > > > > > > > > >>>> > > > > > > > > I believe
> we
> > need to address that as part of this > > >>>> discussion. > > >>>> > > >
> >
> > > > > > > > >>>> > > > > > > > > Cheers, > > >>>> > > > > > > > >
> Dmitri. >
> > > >>>> > > > > > > > > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at
> > 9:29 PM Yufei Gu < > > >>>> flyrain000@gmail.com > > >>>> > > > > > >>>>
> > > > > > > wrote: > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >
> The
> > MetaStore SPI is currently defined with the > > >>>> idea that > > >>>>
> > >
> > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>> >
> >
> > > > > > > > > means one atomic change. > > >>>> > > > > > > > > > > >
> >>>>
> > > > > > > > > > > If the "MetaStore SPI" refers to the interface > >
> >>>> >
> > > > > > > PolarisMetaStoreManager, I > > >>>> > > > > > > > > > don't
> think
> > we've ever state each method to be > > >>>> atomic. We did > > >>>> > >
> > >
> > > clarify > > >>>> > > > > > > > > > atomicity[1] in the interface
> > BasePersistence > > though. > > >>>> > > > > > > > > > > > >>>> > > > >
> > >
> > > > > > > >>>> > > > > > > > > > 1. > > >>>> > > > > > > > > > > > >>>>
> > >
> > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> > >
> >
> > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > >
> >
> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > >
> >
> > > > > > > >>>> > > > > > > > > > Yufei > > >>>> > > > > > > > > > > >
> >>>>
> > > > > > > > > > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at
> > 11:27 AM Dmitri > > Bourlatchkov < > > >>>> > > > > > > dimas@apache.org
> >
> > > > >>>> > > > > > > > > > wrote: > > >>>> > > > > > > > > > > > >>>> >
> > >
> > > > > > > > > Hi Yufei, > > >>>> > > > > > > > > > > > > >>>> > > > > >
> > >
> > > > > I agree that JDBC transactions must be handled > > more > > >>>> >
> >
> > > explicitly. > > >>>> > > > > > > > > However, > > >>>> > > > > > > > >
> >
> > > I'm not sure that simply moving to > > >>>> > > > > > >
> > TransactionalMetaStoreManagerImpl is > > >>>> > > > > > > > > > >
> > sufficient. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > The
> > MetaStore SPI is currently defined with the > > >>>> idea that > > >>>>
> > >
> > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>> >
> >
> > > > > > > > > > means one atomic change [1]. The "transactional" > > >>>>
> > MetaStore > > >>>> > > > > impl. > > >>>> > > > > > is > > >>>> > > > >
> > >
> > > > but a > > >>>> > > > > > > > > > > sub-case of that. It cannot alter
> > the high-level > > >>>> contract. > > >>>> > > > > > > > > > > > > >>>>
> > >
> > > > > > > > > > We could add SPI methods having multiple object > > >>>>
> > parameters > > >>>> > > to > > >>>> > > > > > > represent > > >>>> > > >
> >
> > > > > > > > grouped changes, but I am not sure it will be a > > >>>>
> sound
> > > > >>>> > > design. > > >>>> > > > > > This > > >>>> > > > > > > will >
> >
> > >>>> > > > > > > > > > > bloat the interface surfaces and require extra
> > >
> > >>>> impl. effort > > >>>> > > > for > > >>>> > > > > > > each > > >>>>
> > >
> > > > > > > > > > backend type. More importantly, adding multi-arg > > >>>>
> > change > > >>>> > > > methods > > >>>> > > > > > > still > > >>>> > > >
> > >
> > > > > > won't > > >>>> > > > > > > > > > > address the problem of reads
> > being consistent > with > > >>>> writes, > > >>>> > > > > > because > >
> > >>>> > > > > > > each > > >>>> > > > > > > > > > > method call will still
> > be independent regarding > > the > > >>>> data > > >>>> > > > stored > >
> > >>>> > > > > > in > > >>>> > > > > > > the > > >>>> > > > > > > > > > >
> > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > I
> tend
> > to think we need to introduce a "change > > set" > > >>>> or > > >>>> >
> > >
> > "atomic > > >>>> > > > > > > batch" > > >>>> > > > > > > > > > > concept
> to
> > core Persistence and associate each > > REST > > >>>> API > > >>>> > > >
> > request > > >>>> > > > > > > with > > >>>> > > > > > > > > one > > >>>>
> > >
> > > > > > > > > > such change set, which will be committed (or > > rolled
> > >
> > >>>> back) > > >>>> > > at > > >>>> > > > > the > > >>>> > > > > > > end
> of
> > > > >>>> > > > > > > > > > the > > >>>> > > > > > > > > > > request. I
> > believe Ayush mentioned a similar > > >>>> concept in PR > > >>>> > > >
> > 4939 > > >>>> > > > > > > [2]. In > > >>>> > > > > > > > > > > JDBC each
> > change set will naturally be > associated > > >>>> with an > > >>>> > > >
> > RDBMS > > >>>> > > > > > > > > > > transaction. In NoSQL persistence,
> each
> > atomic > > >>>> change set > > >>>> > > > will > > >>>> > > > > be > >
> >>>>
> > > > > > > > > > > > associated with one CAS operation on the > >
> underlying
> > > > >>>> > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > >
> >
> > > > > [1] > > >>>> > > > > > > > > >>>>
> > https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > >
> >>>>
> > > > > > > > > > > > > > >>>> > > > > > > > > > > [2] > > >>>> > > > > >
> > >
> > > >>>>
> https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > Cheers, > >
> >>>>
> > > > > > > > > > > > Dmitri. > > >>>> > > > > > > > > > > > > >>>> > > >
> > >
> > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < > > >>>> > > >
> > flyrain000@gmail.com > > >>>> > > > > > > > >>>> > > > > > > wrote: > >
> > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > Thanks for
> raising
> > this, Dmitri. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > >
> > >
> > These are valid concerns, and they were > already > > >>>> > >
> recognized >
> > > >>>> > > > > when > > >>>> > > > > > > we > > >>>> > > > > > > > > > >
> >
> > introduced JDBC persistence to Polaris. At > that > > >>>> time, we > >
> > >>>> > > > > chose > > >>>> > > > > > > to use > > >>>> > > > > > > > >
> > >
> > > AtomicOperationMetaStoreManager for the JDBC > due > > >>>> to the > >
> > >>>> > > > > > > simplicity. I > > >>>> > > > > > > > > > > think > >
> >>>>
> > > > > > > > > > > > > most of the issues mentioned here can already > be
> >
> > > >>>> > > addressed > > >>>> > > > by > > >>>> > > > > > > > > > > >
> > TransactionalMetaStoreManagerImpl. > > >>>> > > > > > > > > > > > > >
> >>>>
> > > > > > > > > > > > > For example, rename is already wrapped in a > >
> >>>>
> > transaction > > >>>> > > in > > >>>> > > > > > > > > > > >
> > TransactionalMetaStoreManagerImpl [1]. > > Similarly, > > >>>> catalog >
> >
> > >>>> > > > > > > creation, > > >>>> > > > > > > > > > which > > >>>> > >
> >
> > > > > > > > > > involves reading and creating multiple > objects, > >
> >>>>
> > is also > > >>>> > > > > > executed > > >>>> > > > > > > > > > within a
> > >
> > >>>> > > > > > > > > > > > transaction [2]. > > >>>> > > > > > > > > > >
> >
> > > > >>>> > > > > > > > > > > > I see two possible directions: > > >>>> >
> >
> > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > >
> >
> > > > > > > > >>>> > > > > > > > > > > > Modify
> > AtomicOperationMetaStoreManager > > >>>> together with > > >>>> > > the
> > >
> > >>>> > > > > > > > > persistence > > >>>> > > > > > > > > > > > backends
> > (such as JDBC) to provide the > > required > > >>>> > > > > consistency
> > >
> > >>>> > > > > > > > > > guarantees > > >>>> > > > > > > > > > > > for > >
> > >>>> > > > > > > > > > > > specific operations, similar to what > > >>>>
> >
> > > > > > > > > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > >
> >
> > > > > does. > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > >
> >
> > > > > >>>> > > > > > > > > > > > Migrate the persistence backends (such
> as
> > > > >>>> JDBC) to use > > >>>> > > > > > > > > > > >
> > TransactionalMetaStoreManagerImpl directly. > > We > > >>>> may > > >>>>
> >
> > > have > > >>>> > > > to > > >>>> > > > > > > deal > > >>>> > > > > > >
> > >
> > with > > >>>> > > > > > > > > > > > transactional semantic mismatches
> > across > > >>>> different > > >>>> > > > > > persistence > > >>>> > > >
> > >
> > > > > > > backends. > > >>>> > > > > > > > > > > > For example, we would
> > likely avoid using > > JDBC's > > >>>> > > > > > > > > >
> > `runWithinTransaction` > > >>>> > > > > > > > > > > > for > > >>>> > > >
> >
> > > > > > > > > single row updates, which adds additional > > >>>> overhead
> > and > > >>>> > > > > > > complexity > > >>>> > > > > > > > > > > >
> without
> > > > >>>> > > > > > > > > > > > benefits. > > >>>> > > > > > > > > > > >
> > >
> > >>>> > > > > > > > > > > > References: > > >>>> > > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > > >>>> >
> > >
> > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > >
> > >
> > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > >
> >
> > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > >
> >>>> >
> > > > > > >>>> > > > > >>>> > > >
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > > > > >
> >>>> >
> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > >
> > >
> > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> >
> >
> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > >
> > >>>> > > > > > >>>> > > > > >>>> > > >
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > Yufei > >
> > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > >
> >
> > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri > > >>>>
> Bourlatchkov
> > < > > >>>> > > > > > > > > dimas@apache.org> > > >>>> > > > > > > > > >
> >
> > > wrote: > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > >
> Hi
> > all, > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > >
> > Ayush and Prithvi recently contributed a > > couple > > >>>> of > > >>>>
> >
> > > > > > interesting > > >>>> > > > > > > PRs: > > >>>> > > > > > > > > >
> >
> > > > [4939], [5035]. > > >>>> > > > > > > > > > > > > > > >>>> > > > > >
> > >
> > > > > > > It looks like people are starting to > encounter > > >>>> > > >
> > consistency > > >>>> > > > > > > issues > > >>>> > > > > > > > > in > >
> > >>>> > > > > > > > > > > > > JDBC persistence. > > >>>> > > > > > > > >
> > >
> > > > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight
> into
> > the > > >>>> underlying > > >>>> > > > > issues. > > >>>> > > > > > >
> They
> > > > >>>> > > > > > > > > > offer > > >>>> > > > > > > > > > > > >
> > incremental fixes that can work. However, I > > >>>> believe it > > >>>>
> >
> > > is > > >>>> > > > > > time > > >>>> > > > > > > for > > >>>> > > > > >
> >
> > > > > the > > >>>> > > > > > > > > > > > > Polaris community to review
> and
> > improve this > > >>>> area of > > >>>> > > the > > >>>> > > > > > >
> > codebase > > >>>> > > > > > > > > > > > > holistically. > > >>>> > > > >
> >
> > > > > > > > > > > >>>> > > > > > > > > > > > > By this, I mean finding a
> > solution that can > be > > >>>> applied > > >>>> > > to > > >>>> > > > >
> > all > > >>>> > > > > > > > > > > > > persistence backends (in-memory,
> JDBC,
> > > NoSQL) > > >>>> and > > >>>> > > > addresses > > >>>> > > > > > >
> these >
> > > >>>> > > > > > > > > > > > > aspects: > > >>>> > > > > > > > > > > > >
> >
> > > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent >
> > changes > > >>>> where > > >>>> > > the > > >>>> > > > > > > service > >
> > >>>> > > > > > > > > > reads > > >>>> > > > > > > > > > > > > and
> validates
> > current catalog state, then > > >>>> commits a > > >>>> > > > > change >
> >
> > >>>> > > > > > > (e.g. > > >>>> > > > > > > > > > > > > name clashes
> during
> > renames). > > >>>> > > > > > > > > > > > > * Supporting consistent but
> > independent > > changes > > >>>> to RBAC > > >>>> > > > > > grants > >
> >>>>
> > > > > > > > and > > >>>> > > > > > > > > > > > > MetaStore entities. This
> > independence is > > >>>> needed to > > >>>> > > > > support > > >>>> > >
> >
> > > > > > > > > > > external authorizers like OPA and Ranger. > > >>>> > >
> >
> > > > > > > > > > > * Supporting atomic changes across multiple > > >>>>
> > similar > > >>>> > > > > entities. > > >>>> > > > > > > > > > > > > *
> > Supporting authorization-based filtering > of > > >>>> list > > >>>> > >
> >
> > > operations > > >>>> > > > > > > (cf. > > >>>> > > > > > > > > > > > >
> > [4831]). > > >>>> > > > > > > > > > > > > * Supporting credential-vending
> > decisions > that > > >>>> are > > >>>> > > rooted > > >>>> > > > > in > >
> > >>>> > > > > > > the > > >>>> > > > > > > > > > > > > exact state of the
> > catalog. > > >>>> > > > > > > > > > > > > * Supporting server-side
> retries
> > for > transient > > >>>> > > > persistence > > >>>> > > > > > > failures
> >
> > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures).
> >
> > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Please
> > share your comments and ideas. > > >>>> > > > > > > > > > > > > > > >>>>
> >
> > > > > > > > > > > > > [4831] > > >>>>
> > https://github.com/apache/polaris/pull/4831 > > >>>> > > > > > > > > > >
> > > > > > >>>> > > > > > > > > > > > > [4939] > > >>>>
> > https://github.com/apache/polaris/pull/4939 > > >>>> > > > > > > > > > >
> > > > > > >>>> > > > > > > > > > > > > [5035] > > >>>>
> > https://github.com/apache/polaris/pull/5035 > > >>>> > > > > > > > > > >
> > > > > > >>>> > > > > > > > > > > > > Thanks, > > >>>> > > > > > > > > >
> > >
> > > Dmitri > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > >
> >
> > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > >
> >
> > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> >
> > >
> > > > >>>> > > > > >>>> > > >>> > > >
> >
> >
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by EJ Wang <ej...@gmail.com>.
One detail I omitted: the PoC also includes FoundationDB and Spanner
adapters alongside JDBC. They reuse the same business-aware Manager. The
adapters implement generic storage operations without table, catalog or
grant rules.
This exercises both goals: addressing consistency problems such as
concurrent creation at overlapping table locations, and reusing that logic
across databases. The concurrency tests have run on FoundationDB,
PostgreSQL and CockroachDB. Spanner coverage currently uses the emulator,
with three concurrency tests skipped.
These are targeted feasibility checks. Production validation and
integration with Polaris’s existing NoSQL backend remain follow-up work.
-ej
On Mon, Oct 5, 2026 at 1:36 PM EJ Wang <ej...@gmail.com>
wrote:
> Hi Dmitri, all,
>
> Two requests can both check for overlapping table locations, see none, and
> then create conflicting tables. We need the checks and the resulting writes
> to be protected together.
>
> My proposal is to implement these rules once in shared Polaris code (the
> Manager). This business-aware layer checks them in the transaction that
> creates the table. Database adapters would implement a business-agnostic
> interface (Primitives) for generic reads, writes and commit. They would not
> implement table, catalog or grant rules, so adding a database reuses the
> same Polaris behavior.
>
> This PoC assumes the database itself can protect the relevant reads and
> writes in one serializable transaction. Calls to object storage or
> credential services happen outside it.
>
> The Java PoC
> <https://github.com/flyingImer/polaris/tree/prototype/durable-java-primitives/persistence/primitives>
> reuses the existing transactional Manager and tests selected races,
> authorization retries and uncertain commits. It uses a temporary data
> layout and reads more data than we'd want in production. Existing NoSQL
> integration, production tuning and migration remain follow-up work.
>
> I'm not proposing an additional request-wide Persistence Session yet.
> Existing business-aware request code still owns authorization, external
> work and retries.
>
> The PoC allows an already-authorized metadata change to finish. A fresh
> request retry checks authorization again. An uncertain commit stops
> automatic retries and retains files that may already be referenced. These
> policies still need agreement.
>
> Would this separation be a useful starting point?
>
> -ej
>
>
> On Tue, 22 Sep 2026 19:08:49 -0400, Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> Hi EJ, > 1. Which backend assumptions should we design for? [...] I know I
> used the term "transaction" a lot during the call, but the real challenge
> is about establishing an SPI with clear contracts that could be implemented
> by JDBC and NoSQL persistence using the database features natural to each
> model. I am fairly confident that a JDBC impl. is possible with
> Serializable transaction. We should be able to solve all the problems
> raised in this thread. However, that will make in-memory caching impossible
> because the database needs to see all the reads (and writes) in a Tx to be
> able to ensure serializable guarantees. If a cache prevents a read from
> hitting the database, that read (e.g. storage config) will not be a factor
> in serializability. This connects to my earlier question about the
> in-memory cache and JDBC (title: Usefulness of InMemoryEntityCache) Are
> different approaches possible with JDBC? Probably. What is involved? This
> depends on the SPI contract, I think. > 2. How do we protect the data used
> by validation? [...] The point mentioned by Robert previously IIRC, is that
> we need to ground all runtime decisions to a particular state of the
> catalog. If at commit time the state of the catalog in the database shifts
> from what was used during validation, the request has to restart from
> scratch. Current JDBC Persistence does this only for relationships between
> entities and grants and between versions of the same entity. Relationships
> between table locations and storage config are not actualized for the
> purpose of state consistency tracking. From my POV we need to begin by
> formulating this Persistence state in the Persistence SPI terms. Then,
> think how we can implement that for JDBC. NoSQL Persistence already has a
> per-catalog state (not granular), so wiring the new SPI to NoSQL should not
> be too hard, I hope. The state in the SPI does not have to be global to the
> catalog (as in NoSQL), but it needs to be able to express the idea of which
> elements were engaged during request processing so serve as input to
> Persistence impl. for cosistency guarantees. If Persistence caches data in
> memory (for example), it will have to do so in a way that does not violate
> consistency expectations. From my POV we could begin by defining a
> "Persistence Session" concept. Each request is 1:1 with a session. All
> reads and writes go through the session. The next step would be the JDBC
> impl., which probably needs some brainstorming... but I wonder if we could
> reach a consensus at a high level first. WDYT? Thanks, Dmitri. On Tue, Sep
> 22, 2026 at 5:55 PM EJ Wang <ej...@gmail.com> wrote: >
> Thanks for the recap, Dmitri. I'd like to follow up on the three points I >
> raised during the sync. We didn't have much time to discuss them, so it >
> would help to confirm these before choosing an SPI design. > > 1. Which
> backend assumptions should we design for? > > Within one Polaris server
> instance, do we assume entities, grants and > secrets all use the same
> database backend, or must we support mixing > backends across them? > Must
> every supported persistence implementation provide transactional >
> guarantees? Does the database itself need to provide transactions, or can >
> Polaris build that capability on top of simpler database operations? > >
> I'm happy to work with a homogeneous assumption if that's the intended >
> scope. Existing NoSQL illustrates the second question: it can publish >
> changes to multiple objects under one reference using a single conditional
> > pointer update. > > 2. How do we protect the data used by validation? > >
> Your storage-configuration and location-overlap examples make the problem >
> concrete. Consider these races: > > A table creation passes validation
> against the catalog's allowed locations, > then another request changes
> that configuration before the table is > committed. > Two table-creation
> requests each check for overlapping locations, both see > no overlap, and
> then create tables at conflicting locations. > > Checking only the record
> being written cannot cover these dependencies. The > first also depends on
> the catalog configuration. The second depends on the > absence of another
> overlapping table. > > Could the persistence layer protect these reads
> together with the writes, > using backend-internal versions/tokens or
> database isolation? Shared code > would still need to identify the required
> reads and checks. Could we keep > the mechanism for protecting them inside
> each backend? > > 3. Business rules are duplicated across backend
> implementations > > Today, the TreeMap > < >
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481
> > > > and JDBC > < >
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046
> > > > implementations each load a secret, check its principal, apply the >
> rotation/reset steps and write it back. > > Changing these business rules
> therefore requires keeping multiple > implementations in sync. A fix
> applied to one backend can be missed in > another, causing the same Polaris
> operation to behave differently depending > on the database. > > My
> proposal is to implement each business workflow once, against a common >
> set of storage operations with defined consistency and failure semantics. >
> The shared workflow should not need to know whether those operations are >
> implemented through JDBC transactions, NoSQL reference CAS or another >
> mechanism. Each backend would implement the same storage contract, keeping
> > its execution details internal. A new database backend could then reuse
> the > existing business workflows. > > Do these seem like reasonable
> starting points? We can then evaluate the SPI > against the original
> failure cases on both JDBC and existing NoSQL. > > -ej > > On Fri, Sep 18,
> 2026 at 12:10 PM Dmitri Bourlatchkov <di...@apache.org> > wrote: > > >
> Side note: I just came across [5541], which shows a case where a failed > >
> request leaves persisted side effects. > > > > This can be seen as a coding
> mistake, of course. However, this is about > > executing one request as a
> single, atomic unit. This is something that > > probably needs foundational
> support and coding patterns in Polaris code. > > > > Cheers, > > Dmitri. >
> > [5541] https://github.com/apache/polaris/pull/5541 > > > > On Fri, Sep
> 18, 2026 at 3:02 PM Dmitri Bourlatchkov <di...@apache.org> > > wrote: > >
> > > > Hi All, > > > > > > Here's a recap of the Community Sync discussion
> as I remember it. I'm > > sure > > > my recollection is not complete, so
> please add your thoughts to this > > thread. > > > > > > * Polaris
> traditionally relies on entity version number to ensure > > > consistency
> of changes. > > > > > > This works well for internal RBAC grants, which
> also have version > numbers > > > cross-referenced at the Persistence
> layer. > > > > > > I do not think this works in more general cases like
> validating > locations > > > wrt catalog-level storage configuration. > > >
> > > > Another difficult case involves location overlaps between concurrent
> > > table > > > creations. > > > > > > * We talked about how Persistence
> calls related to RDBMS transactions > (in > > > the JDBC persistence case).
> > > > > > > There were some concerns raised about whether we need to bind >
> Persistence > > > API to transactions explicitly. > > > > > > I think it is
> a valid concern. At the same time, the Persistence API > > > should be
> clear about consistency guarantees across all backend > > >
> implementations. I think this needs more attention now that the matter > of
> > > > milti-entioty changes became prominent in [5035]. > > > > > > I
> personally believe that RDBMS-based Persistence in Polaris has to use > > >
> Serializable transaction isolation. Consequently, the core code needs > to
> > > > provide mechanisms to allow the Persistence implementation to
> properly > > > connect reads and writes from API requests to JDBC
> transactions... > > whether > > > we use "transation" as an explicit terms
> in the SPI or not. > > > > > > Note that currently, the same API request
> performs reads and writes in > > > _separate_ RDBMS transactions (even
> separate connections). > > > > > > * We did not discuss Persistence SPI
> implications in the call, but > > > currently each Persistence call is
> assumed to be one distinct and > atomic > > > change (please correct me if
> I'm wrong). > > > > > > Consequently, supporting new use cases involves
> introducing new SPI > > > methods with many parameters. This can be seen in
> PR [5035]. This tends > > to > > > overcomplicate the SPI. > > > > > >
> Ideally, I think the SPI should be simplified to deal with multi-entity > >
> > and single-entity changes in a coherent manner to avoid any ambiguity > >
> about > > > which method the caller should use. > > > > > > [5035]
> https://github.com/apache/polaris/pull/5035 > > > > > > Cheers, > > >
> Dmitri. > > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <
> dimas@apache.org > > > > > wrote: > > > > > >> Heads up: This discussion
> is on the Community Sync call agenda for > Sept > > >> 17. > > >> > > >>
> Interested parties, please plan to attend, if possible :) > > >> > > >>
> Cheers, > > >> Dmitri. > > >> > > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri
> Bourlatchkov <di...@apache.org> > > >> wrote: > > >> > > >>> Hi JB, > >
> >>> > > >>> I've added an agenda item to the next community sync call for
> this > > (Sept > > >>> 17, 2026). > > >>> > > >>> Re: key points: did you
> mean a real GH discussion or dev email (this > > >>> thread)? Just double
> checking :) I'm fine with either approach. > > >>> > > >>> Cheers, > > >>>
> Dmitri. > > >>> > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste
> Onofré < > jb@nanthrax.net> > > >>> wrote: > > >>> > > >>>> Hi Dmitri, >
> > >>>> > > >>>> I agree on the need to align on these points, though I'm
> not > entirely > > >>>> sure a dedicated meeting is necessary. Let's start
> by using some > time > > >>>> during the next community meeting to discuss
> it. > > >>>> > > >>>> In the meantime, what if we outline the key points in
> a GitHub > > >>>> Discussion first? We can then use that as a reference
> during the > > >>>> meeting. > > >>>> > > >>>> Regards, > > >>>> JB > >
> >>>> > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov < >
> dimas@apache.org > > > > > >>>> wrote: > > >>>> > > > >>>> > Hi All, > >
> >>>> > > > >>>> > My impression from this thread is that it might be time
> for a call > > to > > >>>> > discuss all the related issues and try to
> arrive at a shared > > >>>> implementation > > >>>> > plan. > > >>>> > > >
> >>>> > I know meetings are not ideal, but in this case it might be > > >>>>
> beneficial as a > > >>>> > means for achieving a common understanding of
> the set of problems > > and > > >>>> > priorities related to this thread. A
> dedicated metrics meeting > > worked > > >>>> well > > >>>> > from my POV.
> > > >>>> > > > >>>> > Allocating a time slot in the next community meeting
> might be an > > >>>> option, > > >>>> > although I think we might need the
> full hour in this case. > > >>>> > > > >>>> > WDYT? > > >>>> > > > >>>> >
> Thanks, > > >>>> > Dmitri. > > >>>> > > > >>>> > On Mon, Aug 17, 2026 at
> 6:39 PM Prithvi S < > > >>>> prithvisivasankar@gmail.com> > > >>>> >
> wrote: > > >>>> > > > >>>> > > Hi all, > > >>>> > > > > >>>> > > I rewrote
> https://github.com/apache/polaris/pull/5035, > following > > >>>> the > >
> >>>> > > discussion to focus only on the SPI foundation. > > >>>> > > > >
> >>>> > > What is in the PR now: > > >>>> > > > > >>>> > > - A written
> manager-level consistency contract in > > >>>> > >
> site/content/in-dev/unreleased/persistence-consistency.md. > > >>>> > > -
> CAS-aware EntityMutation / GrantMutation records. > > >>>> > > -
> MetaStoreChangeSet and BasePersistence#commitChangeSet, > with > > >>>>
> backends > > >>>> > > opting in via supportsAtomicMixedCommit(). > > >>>> >
> > - commitChangeSet implementations for the in-memory TreeMap > > >>>>
> backend and > > >>>> > > JDBC. > > >>>> > > -
> AtomicOperationMetaStoreManager and > > >>>>
> TransactionalMetaStoreManagerImpl > > >>>> > > grant/revoke paths now build
> one change set per operation, > > >>>> falling back > > >>>> > > to > >
> >>>> > > individual operations when the backend does not support mixed > >
> >>>> commits. > > >>>> > > - Tests for the change-set API and TreeMap
> atomic commits. > > >>>> > > > > >>>> > > I intentionally left out: > >
> >>>> > > > > >>>> > > - Entity deletes in MetaStoreChangeSet. > > >>>> > >
> - Refactoring createCatalog, dropEntity, and renameEntity to > > use > >
> >>>> change > > >>>> > > sets. > > >>>> > > - NoSQL support for
> commitChangeSet. > > >>>> > > > > >>>> > > I am treating this as Phase 1 so
> the contract and the SPI > > primitive > > >>>> can be > > >>>> > >
> reviewed before the larger operation migrations land. > > >>>> > > > > >>>>
> > > Could you please take a look? In particular, I would like to > know > >
> >>>> whether > > >>>> > > the contract captures the consensus so far, or if
> it needs to > > >>>> address retry > > >>>> > > / external-work
> coordination before we merge this foundation. > > >>>> > > > > >>>> > >
> Thanks, > > >>>> > > Prithvi S > > >>>> > > > > >>>> > > On Mon, Aug 17,
> 2026 at 7:40 PM Dmitri Bourlatchkov < > > >>>> dimas@apache.org> > > >>>>
> > > wrote: > > >>>> > > > > >>>> > > > Hi Robert, > > >>>> > > > > > >>>> >
> > > I agree that the client-visible operation (e.g. REST API > > request) >
> > >>>> > > consists > > >>>> > > > of many distinct phases. The database /
> persistence changes > are > > >>>> just one > > >>>> > > of > > >>>> > > >
> these phases. STS (as an example) is another. My point about > > >>>>
> transactions > > >>>> > > > in JDBC Persistence applies to the former
> (database changes) > > >>>> phase - one > > >>>> > > > retry attempt there
> should ideally involve exactly one > > >>>> transaction. > > >>>> > > > > >
> >>>> > > > We certainly need logic inside Polaris Servers to to >
> coordinate > > >>>> requests > > >>>> > > to > > >>>> > > > various
> external systems depending on outcomes from previous > > >>>> request > >
> >>>> > > > processing phases. > > >>>> > > > > > >>>> > > > I also agree
> that validation based on database state should be > > >>>> repeated > >
> >>>> > > > from scratch if we retry the database changes. > > >>>> > > > >
> > >>>> > > > However, internal RBAC authorization naturally has to happen >
> > >>>> inside the > > >>>> > > > database update phase (read-check-write).
> IIRC, that is part > of > > >>>> the > > >>>> > > > implicit consistency
> guarantees, which were discussed during > > early > > >>>> > > project > >
> >>>> > > > months (but never got written down in full clarity, > > >>>>
> unfortunately). If > > >>>> > > the > > >>>> > > > Authorization check is
> inside that phase for internal RBAC, I > > >>>> suppose it > > >>>> > > >
> will be there for other Authorizers too and that can involve > > >>>>
> external > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly
> expect > > >>>> authorization to be > > >>>> > > > efficient, but I'm not
> sure how quick it can be in practice. > > >>>> There's > > >>>> > > >
> certainly potential for delays in some (perhaps infrequent) > > cases. > >
> >>>> > > > > > >>>> > > > Cheers, > > >>>> > > > Dmitri. > > >>>> > > > > >
> >>>> > > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <
> snazy@snazy.de> > > >>>> wrote: > > >>>> > > > > > >>>> > > > > Hi
> Dmitri, > > >>>> > > > > > > >>>> > > > > I agree that one logical
> operation needs a consistent view > of > > >>>> the whole > > >>>> > > > >
> backend state. > > >>>> > > > > What worries me is treating one long JDBC
> transaction as the > > >>>> boundary > > >>>> > > of > > >>>> > > > > that
> operation and then retrying the complete request if the > > >>>> commit > >
> >>>> > > > fails. > > >>>> > > > > > > >>>> > > > > The request can also
> write metadata to object storage or > call > > >>>> STS. > > >>>> > > > >
> Each system can leave us with an outcome whose certainty we > > >>>> cannot
> > > >>>> > > know. > > >>>> > > > > The database may have committed before
> the connection was > > lost, > > >>>> an > > >>>> > > > > object-store
> write may have succeeded before a timeout, or > STS > > >>>> may have > >
> >>>> > > > > issued credentials before its response was lost. > > >>>> > >
> > > A database rollback cannot undo any of those effects, and > > >>>>
> retrying the > > >>>> > > > > whole request may repeat them. > > >>>> > > >
> > > > >>>> > > > > So I think the contract has to separate the overall >
> operation > > >>>> from each > > >>>> > > > > attempt to commit it. > >
> >>>> > > > > Each database or backend commit attempt should stay short. > >
> >>>> > > > > When we retry, validation and authorization need to use the >
> > >>>> state for > > >>>> > > > that > > >>>> > > > > new attempt, and we
> need clear rules for which external work > > >>>> can be > > >>>> > > >
> reused, > > >>>> > > > > repeated, cleaned up, or reconciled after an
> uncertain > > outcome. > > >>>> > > > > > > >>>> > > > > Some contextual
> state could be useful for carrying that > state. > > >>>> > > > > But the
> JDBC connection and transaction should be an > > >>>> implementation > >
> >>>> > > > detail > > >>>> > > > > inside it, not the boundary of the
> complete operation. > > >>>> > > > > > > >>>> > > > > Cheers, > > >>>> > >
> > > Robert > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > On Mon, Aug
> 10, 2026 at 11:04 PM Dmitri Bourlatchkov < > > >>>> dimas@apache.org > >
> >>>> > > > > > >>>> > > > > wrote: > > >>>> > > > > > > >>>> > > > > > Hi
> All, > > >>>> > > > > > > > >>>> > > > > > The idea of a backend-agnostic
> change-set primitive sounds > > >>>> good. > > >>>> > > > > > > > >>>> > >
> > > > However, I am not sure it is sufficient for all the use > > cases > >
> >>>> we > > >>>> > > > touched > > >>>> > > > > > here. More specifically,
> the state read by validation code > > is > > >>>> not > > >>>> > > > > >
> necessarily reflected in the change set: > > >>>> > > > > > > > >>>> > > >
> > > * Unchanged entities might be considered by validation, > but > > >>>>
> changed > > >>>> > > in > > >>>> > > > a > > >>>> > > > > > parallel
> request. > > >>>> > > > > > * Even for changed entities the state read by
> validation > may > > >>>> differ > > >>>> > > > from > > >>>> > > > > > the
> state being altered. Some validation code talks to the > > >>>> MetaStore >
> > >>>> > > > > > directly, outside of the data produced by the Resolver. >
> > >>>> > > > > > > > >>>> > > > > > I do not think Polaris offers any
> explicit mechanisms > (ATM) > > >>>> to ensure > > >>>> > > > > >
> consistency between these reads and subsequent writes. > > >>>> > > > > > >
> > >>>> > > > > > As far as JDBC goes, running an overarching Transaction >
> > >>>> across all > > >>>> > > > reads > > >>>> > > > > > and writes in the
> same request, at the SERIALIZABLE > > isolation > > >>>> level > > >>>> > >
> in > > >>>> > > > > the > > >>>> > > > > > backing RDBMS could solve the
> problem, I think. > > >>>> > > > > > > > >>>> > > > > > Indeed, such a
> transaction might be long to accommodate > > calls > > >>>> made by > >
> >>>> > > > > > Polaris to external storage, etc. However, is that a > >
> problem? > > >>>> I think > > >>>> > > > the > > >>>> > > > > > alternative
> is for the JDBC Persistence impl. to > "manually" > > >>>> track all > >
> >>>> > > > > reads > > >>>> > > > > > under the same request and redo them
> in the small > > transaction > > >>>> that > > >>>> > > > > persists > >
> >>>> > > > > > the writes. This will also add RDBMS overhead and require >
> > >>>> complex > > >>>> > > code > > >>>> > > > in > > >>>> > > > > >
> Polaris to handle the data properly. > > >>>> > > > > > > > >>>> > > > > >
> Tracking the request-wide transaction can be done only for > > >>>> JDBC >
> > >>>> > > without > > >>>> > > > > > leaking "transaction" concepts to the
> NoSQL Persistence, I > > >>>> think. > > >>>> > > NoSQL > > >>>> > > > > >
> will use other mechanisms to ensure read/write > consistency. > > >>>> > >
> > > > > > >>>> > > > > > Connecting to Robert's email (a parallel branch in
> this > > >>>> discussion), > > >>>> > > > I'd > > >>>> > > > > > like to
> propose this approach: > > >>>> > > > > > > > >>>> > > > > > * Each request
> establishes a "Data Context" > > >>>> > > > > > - In JDBC the Data Context
> corresponds to a JDBC > > Connection > > >>>> + Tx > > >>>> > > > > > - In
> NoSQL the Data Context tracks one or more reference > > >>>> hashes > >
> >>>> > > > > > * All Persistence access in the same request goes through >
> > the > > >>>> same > > >>>> > > Data > > >>>> > > > > > Context > > >>>> >
> > > > > * All Persistence changes are committed once at the end of > > the
> > > >>>> > > request > > >>>> > > > > > - Not all changes have to be
> globally atomic. For > example, > > >>>> changes > > >>>> > > in > > >>>> >
> > > > > Catalogs A and B do not have to be atomic with respect to > > >>>>
> each other. > > >>>> > > > We > > >>>> > > > > > can go deeper into this
> later. This is relevant to NoSQL. > > >>>> > > > > > - Transactional
> backends like JDBC can, of course, > choose > > >>>> to make > > >>>> > >
> all > > >>>> > > > > > changes globally atomic. > > >>>> > > > > > * A
> commit can fail in two main ways: > > >>>> > > > > > - A retriable failure
> like an optimistic lock error or > Tx > > >>>> > > > > serializability > >
> >>>> > > > > > error > > >>>> > > > > > - A non-triable logical error (e.g.
> entity not found) > > >>>> > > > > > * On a retriable error the whole
> request is re-attempted > (as > > >>>> if > > >>>> > > > > resubmitted > >
> >>>> > > > > > by a client) a few times (configurable timeout). > > >>>> >
> > > > > > > >>>> > > > > > WDYT? > > >>>> > > > > > > > >>>> > > > > >
> Cheers, > > >>>> > > > > > Dmitri. > > >>>> > > > > > > > >>>> > > > > > On
> Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré < > > >>>> > >
> jb@nanthrax.net > > >>>> > > > > > > >>>> > > > > > wrote: > > >>>> > > >
> > > > > >>>> > > > > > > Hi all > > >>>> > > > > > > > > >>>> > > > > > > I
> think Robert has a good point. > > >>>> > > > > > > > > >>>> > > > > > > If
> the atomicity guarantee lives only in > BasePersistence, > > >>>> then the
> > > >>>> > > > > > > manager contract can't tell a caller whether the state
> > it > > >>>> read for > > >>>> > > > > > >
> validation/authorization/credential-vending is the same > > >>>> state that
> > > >>>> > > > > > > eventually commits. That's the actual bug class behind
> > the > > >>>> JDBC > > >>>> > > > > > > symptoms (not any single operation
> being non-atomic, but > > the > > >>>> > > contract > > >>>> > > > > > >
> being silent about it. Every operation-specific fix > (like > > >>>> #4939,
> > > >>>> > > > #5035 > > >>>> > > > > > > or #5095) re-answers this
> question locally and it keeps > > >>>> recurring. > > >>>> > > So > > >>>>
> > > > > > > the deliverable should start with a written consistency > >
> >>>> contract at > > >>>> > > > > > > the manager level. > > >>>> > > > > >
> > > > >>>> > > > > > > I'm not sure migrating everyone to > > >>>> > >
> TransactionalMetaStoreManagerImpl > > >>>> > > > > > > is a good idea. It
> would tie the logical change set to a > > DB > > >>>> > > > > > >
> transaction spanning the REST request. It means: > > >>>> > > > > > > - it
> holds a durable transaction open across slow > external > > >>>> work > >
> >>>> > > > > > > (credential vending, OPA, Ranger, ...) > > >>>> > > > > >
> > - it doesn't map to NoSQL > > >>>> > > > > > > - It wraps single-row
> updates in runWiithinTransaction, > > >>>> which is a > > >>>> > > > > >
> overhead > > >>>> > > > > > > > > >>>> > > > > > > So, I think the
> transactional manager isn't a portable > > >>>> target. It's > > >>>> > > >
> > > > "only" one backend's strategy. > > >>>> > > > > > > > > >>>> > > > >
> > > I think Privthi's approach is right. A backend-agnostic > > >>>>
> change-set > > >>>> > > > > > > primitive with a documented fallback is the
> correct > shape. > > >>>> There is > > >>>> > > > > > > one caveat: without
> carrying the original entitiy (not > > just > > >>>> the new > > >>>> > > >
> > > > one) the change set can't express optimistic > concurrency. > > >>>>
> > > > > > > > > >>>> > > > > > > I propose the following multi-steps
> approach: > > >>>> > > > > > > 1. We write the manager-level consistency
> contract as > > >>>> Robert asked. > > >>>> > > > It > > >>>> > > > > > >
> should include the explicit statement that a logical > > change > > >>>>
> set is > > >>>> > > > not > > >>>> > > > > > > a request-scope DB
> transaction. > > >>>> > > > > > > 2. We make Compare And Swap
> (optimistic-concurrency > > pattern) > > >>>> > > baseline > > >>>> > > > >
> > > first-class in EntityMutation before merging the SPI > > >>>> > > > > >
> > 3. Then we refactor > createCatalog/dropEntity/renameEntity. > > >>>> > >
> > > > > > > >>>> > > > > > > Thoughts? > > >>>> > > > > > > > > >>>> > > >
> > > > Regards > > >>>> > > > > > > JB > > >>>> > > > > > > > > >>>> > > > >
> > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp < > > >>>> snazy@snazy.de>
> > > >>>> > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > Hi all, >
> > >>>> > > > > > > > > > >>>> > > > > > > > Yufei's clarification about
> where the atomicity > > guarantee > > >>>> is > > >>>> > > > defined > >
> >>>> > > > > > > seems > > >>>> > > > > > > > important. > > >>>> > > > > >
> > > If it is a property of the lower-level BasePersistence > > >>>>
> contract > > >>>> > > > > rather > > >>>> > > > > > > than > > >>>> > > > >
> > > > the general PolarisMetaStoreManager contract, the > > general > >
> >>>> > > contract > > >>>> > > > > does > > >>>> > > > > > > not > > >>>> >
> > > > > > > tell callers whether the state used for validation, > > >>>> >
> > authorization, > > >>>> > > > or > > >>>> > > > > > > > credential
> vending is consistent with the change that > > >>>> eventually > > >>>> > >
> > > > commits. > > >>>> > > > > > > > > > >>>> > > > > > > > The current
> PRs suggest that operation-specific > > >>>> multi-object > > >>>> > > >
> methods > > >>>> > > > > > can > > >>>> > > > > > > > fix individual cases,
> while leaving the same contract > > >>>> question to > > >>>> > > > > recur
> > > >>>> > > > > > > for > > >>>> > > > > > > > each new case. > > >>>> > >
> > > > > > > > >>>> > > > > > > > I would also avoid defining a logical
> change set as a > > >>>> database > > >>>> > > > > > > transaction > > >>>>
> > > > > > > > around an entire REST request. > > >>>> > > > > > > > That
> would tie the contract to the backend and > database, > > >>>> and > > >>>>
> > > could > > >>>> > > > > keep > > >>>> > > > > > > the > > >>>> > > > > >
> > > durable attempt open across slow external work. > > >>>> > > > > > > >
> > > >>>> > > > > > > > So is the choice really between the two current >
> manager > > >>>> > > > > > implementations, > > >>>> > > > > > > or > >
> >>>> > > > > > > > do we first need to revisit the boundary and >
> guarantees > > >>>> exposed > > >>>> > > to > > >>>> > > > > > their > >
> >>>> > > > > > > > callers? > > >>>> > > > > > > > > > >>>> > > > > > > >
> Cheers, > > >>>> > > > > > > > Robert > > >>>> > > > > > > > > > >>>> > > >
> > > > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri
> Bourlatchkov < > > >>>> > > > > dimas@apache.org > > >>>> > > > > > > > >
> >>>> > > > > > > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > >
> Hi Yufei, > > >>>> > > > > > > > > > > >>>> > > > > > > > > Thanks for the
> link. I stand corrected. The "one > > atomic > > >>>> change > > >>>> > > >
> per > > >>>> > > > > > > method" > > >>>> > > > > > > > > contract as
> defined in javadoc does apply only to > > >>>> > > BasePersistence > > >>>>
> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates > > to
> > > >>>> > > > > > > BasePersistence). > > >>>> > > > > > > > > > > >>>> >
> > > > > > > > Note that all non-test Persistence implementations > in > >
> >>>> the > > >>>> > > Polaris > > >>>> > > > > > > codebase > > >>>> > > >
> > > > > > extend those classes (which is probably why I was > > >>>>
> confused > > >>>> > > about > > >>>> > > > > > > atomicity > > >>>> > > > >
> > > > > expectations). > > >>>> > > > > > > > > > > >>>> > > > > > > > >
> However, this creates a gap in the Persistence SPI > > >>>> > >
> specification. > > >>>> > > > > If > > >>>> > > > > > > > > other
> PolarisMetaStoreManager implementations do not > > >>>> have to > > >>>> >
> > > comply > > >>>> > > > > > > with > > >>>> > > > > > > > > this
> principle, it will create a conceptual > difficulty > > >>>> at call > >
> >>>> > > > > sites. > > >>>> > > > > > > How > > >>>> > > > > > > > > can
> PolarisMetaStoreManager callers reason about > > >>>> consistency > > >>>>
> > > and > > >>>> > > > > > > durability > > >>>> > > > > > > > > behaviours
> in general? > > >>>> > > > > > > > > > > >>>> > > > > > > > > I believe we
> need to address that as part of this > > >>>> discussion. > > >>>> > > > >
> > > > > > > >>>> > > > > > > > > Cheers, > > >>>> > > > > > > > > Dmitri. >
> > >>>> > > > > > > > > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at
> 9:29 PM Yufei Gu < > > >>>> flyrain000@gmail.com > > >>>> > > > > > >>>>
> > > > > > wrote: > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > The
> MetaStore SPI is currently defined with the > > >>>> idea that > > >>>> > >
> one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>> > >
> > > > > > > > means one atomic change. > > >>>> > > > > > > > > > > > >>>>
> > > > > > > > > > If the "MetaStore SPI" refers to the interface > > >>>> >
> > > > > > PolarisMetaStoreManager, I > > >>>> > > > > > > > > > don't think
> we've ever state each method to be > > >>>> atomic. We did > > >>>> > > > >
> > clarify > > >>>> > > > > > > > > > atomicity[1] in the interface
> BasePersistence > > though. > > >>>> > > > > > > > > > > > >>>> > > > > > >
> > > > > > >>>> > > > > > > > > > 1. > > >>>> > > > > > > > > > > > >>>> > >
> > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> > > >
> > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > >
> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > >
> > > > > > >>>> > > > > > > > > > Yufei > > >>>> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at
> 11:27 AM Dmitri > > Bourlatchkov < > > >>>> > > > > > > dimas@apache.org>
> > > >>>> > > > > > > > > > wrote: > > >>>> > > > > > > > > > > > >>>> > > >
> > > > > > > > Hi Yufei, > > >>>> > > > > > > > > > > > > >>>> > > > > > > >
> > > > I agree that JDBC transactions must be handled > > more > > >>>> > >
> > explicitly. > > >>>> > > > > > > > > However, > > >>>> > > > > > > > > >
> > I'm not sure that simply moving to > > >>>> > > > > > >
> TransactionalMetaStoreManagerImpl is > > >>>> > > > > > > > > > >
> sufficient. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > The
> MetaStore SPI is currently defined with the > > >>>> idea that > > >>>> > >
> one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>> > >
> > > > > > > > > means one atomic change [1]. The "transactional" > > >>>>
> MetaStore > > >>>> > > > > impl. > > >>>> > > > > > is > > >>>> > > > > > >
> > > but a > > >>>> > > > > > > > > > > sub-case of that. It cannot alter
> the high-level > > >>>> contract. > > >>>> > > > > > > > > > > > > >>>> > >
> > > > > > > > > We could add SPI methods having multiple object > > >>>>
> parameters > > >>>> > > to > > >>>> > > > > > > represent > > >>>> > > > >
> > > > > > > grouped changes, but I am not sure it will be a > > >>>> sound
> > > >>>> > > design. > > >>>> > > > > > This > > >>>> > > > > > > will > >
> >>>> > > > > > > > > > > bloat the interface surfaces and require extra > >
> >>>> impl. effort > > >>>> > > > for > > >>>> > > > > > > each > > >>>> > >
> > > > > > > > > backend type. More importantly, adding multi-arg > > >>>>
> change > > >>>> > > > methods > > >>>> > > > > > > still > > >>>> > > > > >
> > > > > won't > > >>>> > > > > > > > > > > address the problem of reads
> being consistent > with > > >>>> writes, > > >>>> > > > > > because > >
> >>>> > > > > > > each > > >>>> > > > > > > > > > > method call will still
> be independent regarding > > the > > >>>> data > > >>>> > > > stored > >
> >>>> > > > > > in > > >>>> > > > > > > the > > >>>> > > > > > > > > > >
> database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > I tend
> to think we need to introduce a "change > > set" > > >>>> or > > >>>> > > >
> "atomic > > >>>> > > > > > > batch" > > >>>> > > > > > > > > > > concept to
> core Persistence and associate each > > REST > > >>>> API > > >>>> > > >
> request > > >>>> > > > > > > with > > >>>> > > > > > > > > one > > >>>> > >
> > > > > > > > > such change set, which will be committed (or > > rolled > >
> >>>> back) > > >>>> > > at > > >>>> > > > > the > > >>>> > > > > > > end of
> > > >>>> > > > > > > > > > the > > >>>> > > > > > > > > > > request. I
> believe Ayush mentioned a similar > > >>>> concept in PR > > >>>> > > >
> 4939 > > >>>> > > > > > > [2]. In > > >>>> > > > > > > > > > > JDBC each
> change set will naturally be > associated > > >>>> with an > > >>>> > > >
> RDBMS > > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each
> atomic > > >>>> change set > > >>>> > > > will > > >>>> > > > > be > > >>>>
> > > > > > > > > > > associated with one CAS operation on the > > underlying
> > > >>>> > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > >
> > > > [1] > > >>>> > > > > > > > > >>>>
> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > > >>>>
> > > > > > > > > > > > > >>>> > > > > > > > > > > [2] > > >>>> > > > > > > >
> > >>>> https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > Cheers, > > >>>>
> > > > > > > > > > > Dmitri. > > >>>> > > > > > > > > > > > > >>>> > > > > >
> > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < > > >>>> > > >
> flyrain000@gmail.com > > >>>> > > > > > > > >>>> > > > > > > wrote: > >
> >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > Thanks for raising
> this, Dmitri. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > >
> These are valid concerns, and they were > already > > >>>> > > recognized >
> > >>>> > > > > when > > >>>> > > > > > > we > > >>>> > > > > > > > > > > >
> introduced JDBC persistence to Polaris. At > that > > >>>> time, we > >
> >>>> > > > > chose > > >>>> > > > > > > to use > > >>>> > > > > > > > > > >
> > AtomicOperationMetaStoreManager for the JDBC > due > > >>>> to the > >
> >>>> > > > > > > simplicity. I > > >>>> > > > > > > > > > > think > > >>>>
> > > > > > > > > > > > most of the issues mentioned here can already > be >
> > >>>> > > addressed > > >>>> > > > by > > >>>> > > > > > > > > > > >
> TransactionalMetaStoreManagerImpl. > > >>>> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > For example, rename is already wrapped in a > > >>>>
> transaction > > >>>> > > in > > >>>> > > > > > > > > > > >
> TransactionalMetaStoreManagerImpl [1]. > > Similarly, > > >>>> catalog > >
> >>>> > > > > > > creation, > > >>>> > > > > > > > > > which > > >>>> > > >
> > > > > > > > > involves reading and creating multiple > objects, > > >>>>
> is also > > >>>> > > > > > executed > > >>>> > > > > > > > > > within a > >
> >>>> > > > > > > > > > > > transaction [2]. > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > I see two possible directions: > > >>>> > >
> > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > >
> > > > > > > >>>> > > > > > > > > > > > Modify
> AtomicOperationMetaStoreManager > > >>>> together with > > >>>> > > the > >
> >>>> > > > > > > > > persistence > > >>>> > > > > > > > > > > > backends
> (such as JDBC) to provide the > > required > > >>>> > > > > consistency > >
> >>>> > > > > > > > > > guarantees > > >>>> > > > > > > > > > > > for > >
> >>>> > > > > > > > > > > > specific operations, similar to what > > >>>> >
> > > > > > > > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > > >
> > > > does. > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > Migrate the persistence backends (such as
> > > >>>> JDBC) to use > > >>>> > > > > > > > > > > >
> TransactionalMetaStoreManagerImpl directly. > > We > > >>>> may > > >>>> >
> > have > > >>>> > > > to > > >>>> > > > > > > deal > > >>>> > > > > > > > >
> with > > >>>> > > > > > > > > > > > transactional semantic mismatches
> across > > >>>> different > > >>>> > > > > > persistence > > >>>> > > > > >
> > > > > > backends. > > >>>> > > > > > > > > > > > For example, we would
> likely avoid using > > JDBC's > > >>>> > > > > > > > > >
> `runWithinTransaction` > > >>>> > > > > > > > > > > > for > > >>>> > > > >
> > > > > > > > single row updates, which adds additional > > >>>> overhead
> and > > >>>> > > > > > > complexity > > >>>> > > > > > > > > > > > without
> > > >>>> > > > > > > > > > > > benefits. > > >>>> > > > > > > > > > > > > >
> >>>> > > > > > > > > > > > References: > > >>>> > > > > > > > > > > > > >
> >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > > >>>> > > >
> > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > >
> > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> >
> > > > > >>>> > > > > >>>> > > >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > > > > > >>>> >
> > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > >
> > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > >
> >>>> > > > > > >>>> > > > > >>>> > > >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > Yufei > >
> >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > >
> > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri > > >>>> Bourlatchkov
> < > > >>>> > > > > > > > > dimas@apache.org> > > >>>> > > > > > > > > > >
> > wrote: > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Hi
> all, > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > >
> Ayush and Prithvi recently contributed a > > couple > > >>>> of > > >>>> >
> > > > > interesting > > >>>> > > > > > > PRs: > > >>>> > > > > > > > > > >
> > > [4939], [5035]. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > >
> > > > > > It looks like people are starting to > encounter > > >>>> > > >
> consistency > > >>>> > > > > > > issues > > >>>> > > > > > > > > in > >
> >>>> > > > > > > > > > > > > JDBC persistence. > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight into
> the > > >>>> underlying > > >>>> > > > > issues. > > >>>> > > > > > > They
> > > >>>> > > > > > > > > > offer > > >>>> > > > > > > > > > > > >
> incremental fixes that can work. However, I > > >>>> believe it > > >>>> >
> > is > > >>>> > > > > > time > > >>>> > > > > > > for > > >>>> > > > > > >
> > > > the > > >>>> > > > > > > > > > > > > Polaris community to review and
> improve this > > >>>> area of > > >>>> > > the > > >>>> > > > > > >
> codebase > > >>>> > > > > > > > > > > > > holistically. > > >>>> > > > > >
> > > > > > > > > > >>>> > > > > > > > > > > > > By this, I mean finding a
> solution that can > be > > >>>> applied > > >>>> > > to > > >>>> > > > >
> all > > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC,
> > NoSQL) > > >>>> and > > >>>> > > > addresses > > >>>> > > > > > > these >
> > >>>> > > > > > > > > > > > > aspects: > > >>>> > > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent >
> changes > > >>>> where > > >>>> > > the > > >>>> > > > > > > service > >
> >>>> > > > > > > > > > reads > > >>>> > > > > > > > > > > > > and validates
> current catalog state, then > > >>>> commits a > > >>>> > > > > change > >
> >>>> > > > > > > (e.g. > > >>>> > > > > > > > > > > > > name clashes during
> renames). > > >>>> > > > > > > > > > > > > * Supporting consistent but
> independent > > changes > > >>>> to RBAC > > >>>> > > > > > grants > > >>>>
> > > > > > > and > > >>>> > > > > > > > > > > > > MetaStore entities. This
> independence is > > >>>> needed to > > >>>> > > > > support > > >>>> > > >
> > > > > > > > > > external authorizers like OPA and Ranger. > > >>>> > > >
> > > > > > > > > > * Supporting atomic changes across multiple > > >>>>
> similar > > >>>> > > > > entities. > > >>>> > > > > > > > > > > > > *
> Supporting authorization-based filtering > of > > >>>> list > > >>>> > > >
> > operations > > >>>> > > > > > > (cf. > > >>>> > > > > > > > > > > > >
> [4831]). > > >>>> > > > > > > > > > > > > * Supporting credential-vending
> decisions > that > > >>>> are > > >>>> > > rooted > > >>>> > > > > in > >
> >>>> > > > > > > the > > >>>> > > > > > > > > > > > > exact state of the
> catalog. > > >>>> > > > > > > > > > > > > * Supporting server-side retries
> for > transient > > >>>> > > > persistence > > >>>> > > > > > > failures >
> > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures). >
> > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Please
> share your comments and ideas. > > >>>> > > > > > > > > > > > > > > >>>> >
> > > > > > > > > > > > [4831] > > >>>>
> https://github.com/apache/polaris/pull/4831 > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > [4939] > > >>>>
> https://github.com/apache/polaris/pull/4939 > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > [5035] > > >>>>
> https://github.com/apache/polaris/pull/5035 > > >>>> > > > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > Thanks, > > >>>> > > > > > > > > > > >
> > Dmitri > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > >
> > > >>>> > > > > >>>> > > >>> > > >
>
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by EJ Wang <ej...@gmail.com>.
Hi Dmitri, all,
Two requests can both check for overlapping table locations, see none, and
then create conflicting tables. We need the checks and the resulting writes
to be protected together.
My proposal is to implement these rules once in shared Polaris code (the
Manager). This business-aware layer checks them in the transaction that
creates the table. Database adapters would implement a business-agnostic
interface (Primitives) for generic reads, writes and commit. They would not
implement table, catalog or grant rules, so adding a database reuses the
same Polaris behavior.
This PoC assumes the database itself can protect the relevant reads and
writes in one serializable transaction. Calls to object storage or
credential services happen outside it.
The Java PoC
<https://github.com/flyingImer/polaris/tree/prototype/durable-java-primitives/persistence/primitives>
reuses the existing transactional Manager and tests selected races,
authorization retries and uncertain commits. It uses a temporary data
layout and reads more data than we'd want in production. Existing NoSQL
integration, production tuning and migration remain follow-up work.
I'm not proposing an additional request-wide Persistence Session yet.
Existing business-aware request code still owns authorization, external
work and retries.
The PoC allows an already-authorized metadata change to finish. A fresh
request retry checks authorization again. An uncertain commit stops
automatic retries and retains files that may already be referenced. These
policies still need agreement.
Would this separation be a useful starting point?
-ej
On Tue, 22 Sep 2026 19:08:49 -0400, Dmitri Bourlatchkov <di...@apache.org>
wrote:
Hi EJ, > 1. Which backend assumptions should we design for? [...] I know I
used the term "transaction" a lot during the call, but the real challenge
is about establishing an SPI with clear contracts that could be implemented
by JDBC and NoSQL persistence using the database features natural to each
model. I am fairly confident that a JDBC impl. is possible with
Serializable transaction. We should be able to solve all the problems
raised in this thread. However, that will make in-memory caching impossible
because the database needs to see all the reads (and writes) in a Tx to be
able to ensure serializable guarantees. If a cache prevents a read from
hitting the database, that read (e.g. storage config) will not be a factor
in serializability. This connects to my earlier question about the
in-memory cache and JDBC (title: Usefulness of InMemoryEntityCache) Are
different approaches possible with JDBC? Probably. What is involved? This
depends on the SPI contract, I think. > 2. How do we protect the data used
by validation? [...] The point mentioned by Robert previously IIRC, is that
we need to ground all runtime decisions to a particular state of the
catalog. If at commit time the state of the catalog in the database shifts
from what was used during validation, the request has to restart from
scratch. Current JDBC Persistence does this only for relationships between
entities and grants and between versions of the same entity. Relationships
between table locations and storage config are not actualized for the
purpose of state consistency tracking. From my POV we need to begin by
formulating this Persistence state in the Persistence SPI terms. Then,
think how we can implement that for JDBC. NoSQL Persistence already has a
per-catalog state (not granular), so wiring the new SPI to NoSQL should not
be too hard, I hope. The state in the SPI does not have to be global to the
catalog (as in NoSQL), but it needs to be able to express the idea of which
elements were engaged during request processing so serve as input to
Persistence impl. for cosistency guarantees. If Persistence caches data in
memory (for example), it will have to do so in a way that does not violate
consistency expectations. From my POV we could begin by defining a
"Persistence Session" concept. Each request is 1:1 with a session. All
reads and writes go through the session. The next step would be the JDBC
impl., which probably needs some brainstorming... but I wonder if we could
reach a consensus at a high level first. WDYT? Thanks, Dmitri. On Tue, Sep
22, 2026 at 5:55 PM EJ Wang <ej...@gmail.com> wrote: >
Thanks for the recap, Dmitri. I'd like to follow up on the three points I >
raised during the sync. We didn't have much time to discuss them, so it >
would help to confirm these before choosing an SPI design. > > 1. Which
backend assumptions should we design for? > > Within one Polaris server
instance, do we assume entities, grants and > secrets all use the same
database backend, or must we support mixing > backends across them? > Must
every supported persistence implementation provide transactional >
guarantees? Does the database itself need to provide transactions, or can >
Polaris build that capability on top of simpler database operations? > >
I'm happy to work with a homogeneous assumption if that's the intended >
scope. Existing NoSQL illustrates the second question: it can publish >
changes to multiple objects under one reference using a single conditional
> pointer update. > > 2. How do we protect the data used by validation? > >
Your storage-configuration and location-overlap examples make the problem >
concrete. Consider these races: > > A table creation passes validation
against the catalog's allowed locations, > then another request changes
that configuration before the table is > committed. > Two table-creation
requests each check for overlapping locations, both see > no overlap, and
then create tables at conflicting locations. > > Checking only the record
being written cannot cover these dependencies. The > first also depends on
the catalog configuration. The second depends on the > absence of another
overlapping table. > > Could the persistence layer protect these reads
together with the writes, > using backend-internal versions/tokens or
database isolation? Shared code > would still need to identify the required
reads and checks. Could we keep > the mechanism for protecting them inside
each backend? > > 3. Business rules are duplicated across backend
implementations > > Today, the TreeMap > < >
https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481
> > > and JDBC > < >
https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046
> > > implementations each load a secret, check its principal, apply the >
rotation/reset steps and write it back. > > Changing these business rules
therefore requires keeping multiple > implementations in sync. A fix
applied to one backend can be missed in > another, causing the same Polaris
operation to behave differently depending > on the database. > > My
proposal is to implement each business workflow once, against a common >
set of storage operations with defined consistency and failure semantics. >
The shared workflow should not need to know whether those operations are >
implemented through JDBC transactions, NoSQL reference CAS or another >
mechanism. Each backend would implement the same storage contract, keeping
> its execution details internal. A new database backend could then reuse
the > existing business workflows. > > Do these seem like reasonable
starting points? We can then evaluate the SPI > against the original
failure cases on both JDBC and existing NoSQL. > > -ej > > On Fri, Sep 18,
2026 at 12:10 PM Dmitri Bourlatchkov <di...@apache.org> > wrote: > > > Side
note: I just came across [5541], which shows a case where a failed > >
request leaves persisted side effects. > > > > This can be seen as a coding
mistake, of course. However, this is about > > executing one request as a
single, atomic unit. This is something that > > probably needs foundational
support and coding patterns in Polaris code. > > > > Cheers, > > Dmitri. >
> [5541] https://github.com/apache/polaris/pull/5541 > > > > On Fri, Sep
18, 2026 at 3:02 PM Dmitri Bourlatchkov <di...@apache.org> > > wrote: > > >
> > Hi All, > > > > > > Here's a recap of the Community Sync discussion as
I remember it. I'm > > sure > > > my recollection is not complete, so
please add your thoughts to this > > thread. > > > > > > * Polaris
traditionally relies on entity version number to ensure > > > consistency
of changes. > > > > > > This works well for internal RBAC grants, which
also have version > numbers > > > cross-referenced at the Persistence
layer. > > > > > > I do not think this works in more general cases like
validating > locations > > > wrt catalog-level storage configuration. > > >
> > > Another difficult case involves location overlaps between concurrent
> > table > > > creations. > > > > > > * We talked about how Persistence
calls related to RDBMS transactions > (in > > > the JDBC persistence case).
> > > > > > There were some concerns raised about whether we need to bind >
Persistence > > > API to transactions explicitly. > > > > > > I think it is
a valid concern. At the same time, the Persistence API > > > should be
clear about consistency guarantees across all backend > > >
implementations. I think this needs more attention now that the matter > of
> > > milti-entioty changes became prominent in [5035]. > > > > > > I
personally believe that RDBMS-based Persistence in Polaris has to use > > >
Serializable transaction isolation. Consequently, the core code needs > to
> > > provide mechanisms to allow the Persistence implementation to
properly > > > connect reads and writes from API requests to JDBC
transactions... > > whether > > > we use "transation" as an explicit terms
in the SPI or not. > > > > > > Note that currently, the same API request
performs reads and writes in > > > _separate_ RDBMS transactions (even
separate connections). > > > > > > * We did not discuss Persistence SPI
implications in the call, but > > > currently each Persistence call is
assumed to be one distinct and > atomic > > > change (please correct me if
I'm wrong). > > > > > > Consequently, supporting new use cases involves
introducing new SPI > > > methods with many parameters. This can be seen in
PR [5035]. This tends > > to > > > overcomplicate the SPI. > > > > > >
Ideally, I think the SPI should be simplified to deal with multi-entity > >
> and single-entity changes in a coherent manner to avoid any ambiguity > >
about > > > which method the caller should use. > > > > > > [5035]
https://github.com/apache/polaris/pull/5035 > > > > > > Cheers, > > >
Dmitri. > > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <
dimas@apache.org > > > > > wrote: > > > > > >> Heads up: This discussion is
on the Community Sync call agenda for > Sept > > >> 17. > > >> > > >>
Interested parties, please plan to attend, if possible :) > > >> > > >>
Cheers, > > >> Dmitri. > > >> > > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri
Bourlatchkov <di...@apache.org> > > >> wrote: > > >> > > >>> Hi JB, > > >>>
> > >>> I've added an agenda item to the next community sync call for this
> > (Sept > > >>> 17, 2026). > > >>> > > >>> Re: key points: did you mean a
real GH discussion or dev email (this > > >>> thread)? Just double checking
:) I'm fine with either approach. > > >>> > > >>> Cheers, > > >>> Dmitri. >
> >>> > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré < >
jb@nanthrax.net> > > >>> wrote: > > >>> > > >>>> Hi Dmitri, > > >>>> > >
>>>> I agree on the need to align on these points, though I'm not >
entirely > > >>>> sure a dedicated meeting is necessary. Let's start by
using some > time > > >>>> during the next community meeting to discuss it.
> > >>>> > > >>>> In the meantime, what if we outline the key points in a
GitHub > > >>>> Discussion first? We can then use that as a reference
during the > > >>>> meeting. > > >>>> > > >>>> Regards, > > >>>> JB > >
>>>> > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov < >
dimas@apache.org > > > > > >>>> wrote: > > >>>> > > > >>>> > Hi All, > >
>>>> > > > >>>> > My impression from this thread is that it might be time
for a call > > to > > >>>> > discuss all the related issues and try to
arrive at a shared > > >>>> implementation > > >>>> > plan. > > >>>> > > >
>>>> > I know meetings are not ideal, but in this case it might be > > >>>>
beneficial as a > > >>>> > means for achieving a common understanding of
the set of problems > > and > > >>>> > priorities related to this thread. A
dedicated metrics meeting > > worked > > >>>> well > > >>>> > from my POV.
> > >>>> > > > >>>> > Allocating a time slot in the next community meeting
might be an > > >>>> option, > > >>>> > although I think we might need the
full hour in this case. > > >>>> > > > >>>> > WDYT? > > >>>> > > > >>>> >
Thanks, > > >>>> > Dmitri. > > >>>> > > > >>>> > On Mon, Aug 17, 2026 at
6:39 PM Prithvi S < > > >>>> prithvisivasankar@gmail.com> > > >>>> > wrote:
> > >>>> > > > >>>> > > Hi all, > > >>>> > > > > >>>> > > I rewrote
https://github.com/apache/polaris/pull/5035, > following > > >>>> the > >
>>>> > > discussion to focus only on the SPI foundation. > > >>>> > > > >
>>>> > > What is in the PR now: > > >>>> > > > > >>>> > > - A written
manager-level consistency contract in > > >>>> > >
site/content/in-dev/unreleased/persistence-consistency.md. > > >>>> > > -
CAS-aware EntityMutation / GrantMutation records. > > >>>> > > -
MetaStoreChangeSet and BasePersistence#commitChangeSet, > with > > >>>>
backends > > >>>> > > opting in via supportsAtomicMixedCommit(). > > >>>> >
> - commitChangeSet implementations for the in-memory TreeMap > > >>>>
backend and > > >>>> > > JDBC. > > >>>> > > -
AtomicOperationMetaStoreManager and > > >>>>
TransactionalMetaStoreManagerImpl > > >>>> > > grant/revoke paths now build
one change set per operation, > > >>>> falling back > > >>>> > > to > >
>>>> > > individual operations when the backend does not support mixed > >
>>>> commits. > > >>>> > > - Tests for the change-set API and TreeMap
atomic commits. > > >>>> > > > > >>>> > > I intentionally left out: > >
>>>> > > > > >>>> > > - Entity deletes in MetaStoreChangeSet. > > >>>> > >
- Refactoring createCatalog, dropEntity, and renameEntity to > > use > >
>>>> change > > >>>> > > sets. > > >>>> > > - NoSQL support for
commitChangeSet. > > >>>> > > > > >>>> > > I am treating this as Phase 1 so
the contract and the SPI > > primitive > > >>>> can be > > >>>> > >
reviewed before the larger operation migrations land. > > >>>> > > > > >>>>
> > Could you please take a look? In particular, I would like to > know > >
>>>> whether > > >>>> > > the contract captures the consensus so far, or if
it needs to > > >>>> address retry > > >>>> > > / external-work
coordination before we merge this foundation. > > >>>> > > > > >>>> > >
Thanks, > > >>>> > > Prithvi S > > >>>> > > > > >>>> > > On Mon, Aug 17,
2026 at 7:40 PM Dmitri Bourlatchkov < > > >>>> dimas@apache.org> > > >>>> >
> wrote: > > >>>> > > > > >>>> > > > Hi Robert, > > >>>> > > > > > >>>> > >
> I agree that the client-visible operation (e.g. REST API > > request) > >
>>>> > > consists > > >>>> > > > of many distinct phases. The database /
persistence changes > are > > >>>> just one > > >>>> > > of > > >>>> > > >
these phases. STS (as an example) is another. My point about > > >>>>
transactions > > >>>> > > > in JDBC Persistence applies to the former
(database changes) > > >>>> phase - one > > >>>> > > > retry attempt there
should ideally involve exactly one > > >>>> transaction. > > >>>> > > > > >
>>>> > > > We certainly need logic inside Polaris Servers to to >
coordinate > > >>>> requests > > >>>> > > to > > >>>> > > > various
external systems depending on outcomes from previous > > >>>> request > >
>>>> > > > processing phases. > > >>>> > > > > > >>>> > > > I also agree
that validation based on database state should be > > >>>> repeated > >
>>>> > > > from scratch if we retry the database changes. > > >>>> > > > >
> >>>> > > > However, internal RBAC authorization naturally has to happen >
> >>>> inside the > > >>>> > > > database update phase (read-check-write).
IIRC, that is part > of > > >>>> the > > >>>> > > > implicit consistency
guarantees, which were discussed during > > early > > >>>> > > project > >
>>>> > > > months (but never got written down in full clarity, > > >>>>
unfortunately). If > > >>>> > > the > > >>>> > > > Authorization check is
inside that phase for internal RBAC, I > > >>>> suppose it > > >>>> > > >
will be there for other Authorizers too and that can involve > > >>>>
external > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly
expect > > >>>> authorization to be > > >>>> > > > efficient, but I'm not
sure how quick it can be in practice. > > >>>> There's > > >>>> > > >
certainly potential for delays in some (perhaps infrequent) > > cases. > >
>>>> > > > > > >>>> > > > Cheers, > > >>>> > > > Dmitri. > > >>>> > > > > >
>>>> > > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <
snazy@snazy.de> > > >>>> wrote: > > >>>> > > > > > >>>> > > > > Hi Dmitri,
> > >>>> > > > > > > >>>> > > > > I agree that one logical operation needs
a consistent view > of > > >>>> the whole > > >>>> > > > > backend state. >
> >>>> > > > > What worries me is treating one long JDBC transaction as the
> > >>>> boundary > > >>>> > > of > > >>>> > > > > that operation and then
retrying the complete request if the > > >>>> commit > > >>>> > > > fails.
> > >>>> > > > > > > >>>> > > > > The request can also write metadata to
object storage or > call > > >>>> STS. > > >>>> > > > > Each system can
leave us with an outcome whose certainty we > > >>>> cannot > > >>>> > >
know. > > >>>> > > > > The database may have committed before the
connection was > > lost, > > >>>> an > > >>>> > > > > object-store write
may have succeeded before a timeout, or > STS > > >>>> may have > > >>>> >
> > > issued credentials before its response was lost. > > >>>> > > > > A
database rollback cannot undo any of those effects, and > > >>>> retrying
the > > >>>> > > > > whole request may repeat them. > > >>>> > > > > > >
>>>> > > > > So I think the contract has to separate the overall >
operation > > >>>> from each > > >>>> > > > > attempt to commit it. > >
>>>> > > > > Each database or backend commit attempt should stay short. > >
>>>> > > > > When we retry, validation and authorization need to use the >
> >>>> state for > > >>>> > > > that > > >>>> > > > > new attempt, and we
need clear rules for which external work > > >>>> can be > > >>>> > > >
reused, > > >>>> > > > > repeated, cleaned up, or reconciled after an
uncertain > > outcome. > > >>>> > > > > > > >>>> > > > > Some contextual
state could be useful for carrying that > state. > > >>>> > > > > But the
JDBC connection and transaction should be an > > >>>> implementation > >
>>>> > > > detail > > >>>> > > > > inside it, not the boundary of the
complete operation. > > >>>> > > > > > > >>>> > > > > Cheers, > > >>>> > >
> > Robert > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > On Mon, Aug
10, 2026 at 11:04 PM Dmitri Bourlatchkov < > > >>>> dimas@apache.org > >
>>>> > > > > > >>>> > > > > wrote: > > >>>> > > > > > > >>>> > > > > > Hi
All, > > >>>> > > > > > > > >>>> > > > > > The idea of a backend-agnostic
change-set primitive sounds > > >>>> good. > > >>>> > > > > > > > >>>> > >
> > > However, I am not sure it is sufficient for all the use > > cases > >
>>>> we > > >>>> > > > touched > > >>>> > > > > > here. More specifically,
the state read by validation code > > is > > >>>> not > > >>>> > > > > >
necessarily reflected in the change set: > > >>>> > > > > > > > >>>> > > >
> > * Unchanged entities might be considered by validation, > but > > >>>>
changed > > >>>> > > in > > >>>> > > > a > > >>>> > > > > > parallel
request. > > >>>> > > > > > * Even for changed entities the state read by
validation > may > > >>>> differ > > >>>> > > > from > > >>>> > > > > > the
state being altered. Some validation code talks to the > > >>>> MetaStore >
> >>>> > > > > > directly, outside of the data produced by the Resolver. >
> >>>> > > > > > > > >>>> > > > > > I do not think Polaris offers any
explicit mechanisms > (ATM) > > >>>> to ensure > > >>>> > > > > >
consistency between these reads and subsequent writes. > > >>>> > > > > > >
> >>>> > > > > > As far as JDBC goes, running an overarching Transaction >
> >>>> across all > > >>>> > > > reads > > >>>> > > > > > and writes in the
same request, at the SERIALIZABLE > > isolation > > >>>> level > > >>>> > >
in > > >>>> > > > > the > > >>>> > > > > > backing RDBMS could solve the
problem, I think. > > >>>> > > > > > > > >>>> > > > > > Indeed, such a
transaction might be long to accommodate > > calls > > >>>> made by > >
>>>> > > > > > Polaris to external storage, etc. However, is that a > >
problem? > > >>>> I think > > >>>> > > > the > > >>>> > > > > > alternative
is for the JDBC Persistence impl. to > "manually" > > >>>> track all > >
>>>> > > > > reads > > >>>> > > > > > under the same request and redo them
in the small > > transaction > > >>>> that > > >>>> > > > > persists > >
>>>> > > > > > the writes. This will also add RDBMS overhead and require >
> >>>> complex > > >>>> > > code > > >>>> > > > in > > >>>> > > > > >
Polaris to handle the data properly. > > >>>> > > > > > > > >>>> > > > > >
Tracking the request-wide transaction can be done only for > > >>>> JDBC >
> >>>> > > without > > >>>> > > > > > leaking "transaction" concepts to the
NoSQL Persistence, I > > >>>> think. > > >>>> > > NoSQL > > >>>> > > > > >
will use other mechanisms to ensure read/write > consistency. > > >>>> > >
> > > > > >>>> > > > > > Connecting to Robert's email (a parallel branch in
this > > >>>> discussion), > > >>>> > > > I'd > > >>>> > > > > > like to
propose this approach: > > >>>> > > > > > > > >>>> > > > > > * Each request
establishes a "Data Context" > > >>>> > > > > > - In JDBC the Data Context
corresponds to a JDBC > > Connection > > >>>> + Tx > > >>>> > > > > > - In
NoSQL the Data Context tracks one or more reference > > >>>> hashes > >
>>>> > > > > > * All Persistence access in the same request goes through >
> the > > >>>> same > > >>>> > > Data > > >>>> > > > > > Context > > >>>> >
> > > > * All Persistence changes are committed once at the end of > > the
> > >>>> > > request > > >>>> > > > > > - Not all changes have to be
globally atomic. For > example, > > >>>> changes > > >>>> > > in > > >>>> >
> > > > Catalogs A and B do not have to be atomic with respect to > > >>>>
each other. > > >>>> > > > We > > >>>> > > > > > can go deeper into this
later. This is relevant to NoSQL. > > >>>> > > > > > - Transactional
backends like JDBC can, of course, > choose > > >>>> to make > > >>>> > >
all > > >>>> > > > > > changes globally atomic. > > >>>> > > > > > * A
commit can fail in two main ways: > > >>>> > > > > > - A retriable failure
like an optimistic lock error or > Tx > > >>>> > > > > serializability > >
>>>> > > > > > error > > >>>> > > > > > - A non-triable logical error (e.g.
entity not found) > > >>>> > > > > > * On a retriable error the whole
request is re-attempted > (as > > >>>> if > > >>>> > > > > resubmitted > >
>>>> > > > > > by a client) a few times (configurable timeout). > > >>>> >
> > > > > > >>>> > > > > > WDYT? > > >>>> > > > > > > > >>>> > > > > >
Cheers, > > >>>> > > > > > Dmitri. > > >>>> > > > > > > > >>>> > > > > > On
Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré < > > >>>> > >
jb@nanthrax.net > > >>>> > > > > > > >>>> > > > > > wrote: > > >>>> > > > >
> > > >>>> > > > > > > Hi all > > >>>> > > > > > > > > >>>> > > > > > > I
think Robert has a good point. > > >>>> > > > > > > > > >>>> > > > > > > If
the atomicity guarantee lives only in > BasePersistence, > > >>>> then the
> > >>>> > > > > > > manager contract can't tell a caller whether the state
> it > > >>>> read for > > >>>> > > > > > >
validation/authorization/credential-vending is the same > > >>>> state that
> > >>>> > > > > > > eventually commits. That's the actual bug class behind
> the > > >>>> JDBC > > >>>> > > > > > > symptoms (not any single operation
being non-atomic, but > > the > > >>>> > > contract > > >>>> > > > > > >
being silent about it. Every operation-specific fix > (like > > >>>> #4939,
> > >>>> > > > #5035 > > >>>> > > > > > > or #5095) re-answers this
question locally and it keeps > > >>>> recurring. > > >>>> > > So > > >>>>
> > > > > > the deliverable should start with a written consistency > >
>>>> contract at > > >>>> > > > > > > the manager level. > > >>>> > > > > >
> > > >>>> > > > > > > I'm not sure migrating everyone to > > >>>> > >
TransactionalMetaStoreManagerImpl > > >>>> > > > > > > is a good idea. It
would tie the logical change set to a > > DB > > >>>> > > > > > >
transaction spanning the REST request. It means: > > >>>> > > > > > > - it
holds a durable transaction open across slow > external > > >>>> work > >
>>>> > > > > > > (credential vending, OPA, Ranger, ...) > > >>>> > > > > >
> - it doesn't map to NoSQL > > >>>> > > > > > > - It wraps single-row
updates in runWiithinTransaction, > > >>>> which is a > > >>>> > > > > >
overhead > > >>>> > > > > > > > > >>>> > > > > > > So, I think the
transactional manager isn't a portable > > >>>> target. It's > > >>>> > > >
> > > "only" one backend's strategy. > > >>>> > > > > > > > > >>>> > > > >
> > I think Privthi's approach is right. A backend-agnostic > > >>>>
change-set > > >>>> > > > > > > primitive with a documented fallback is the
correct > shape. > > >>>> There is > > >>>> > > > > > > one caveat: without
carrying the original entitiy (not > > just > > >>>> the new > > >>>> > > >
> > > one) the change set can't express optimistic > concurrency. > > >>>>
> > > > > > > > >>>> > > > > > > I propose the following multi-steps
approach: > > >>>> > > > > > > 1. We write the manager-level consistency
contract as > > >>>> Robert asked. > > >>>> > > > It > > >>>> > > > > > >
should include the explicit statement that a logical > > change > > >>>>
set is > > >>>> > > > not > > >>>> > > > > > > a request-scope DB
transaction. > > >>>> > > > > > > 2. We make Compare And Swap
(optimistic-concurrency > > pattern) > > >>>> > > baseline > > >>>> > > > >
> > first-class in EntityMutation before merging the SPI > > >>>> > > > > >
> 3. Then we refactor > createCatalog/dropEntity/renameEntity. > > >>>> > >
> > > > > > >>>> > > > > > > Thoughts? > > >>>> > > > > > > > > >>>> > > >
> > > Regards > > >>>> > > > > > > JB > > >>>> > > > > > > > > >>>> > > > >
> > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp < > > >>>> snazy@snazy.de>
> > >>>> > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > Hi all, >
> >>>> > > > > > > > > > >>>> > > > > > > > Yufei's clarification about
where the atomicity > > guarantee > > >>>> is > > >>>> > > > defined > >
>>>> > > > > > > seems > > >>>> > > > > > > > important. > > >>>> > > > > >
> > If it is a property of the lower-level BasePersistence > > >>>>
contract > > >>>> > > > > rather > > >>>> > > > > > > than > > >>>> > > > >
> > > the general PolarisMetaStoreManager contract, the > > general > >
>>>> > > contract > > >>>> > > > > does > > >>>> > > > > > > not > > >>>> >
> > > > > > tell callers whether the state used for validation, > > >>>> >
> authorization, > > >>>> > > > or > > >>>> > > > > > > > credential
vending is consistent with the change that > > >>>> eventually > > >>>> > >
> > > commits. > > >>>> > > > > > > > > > >>>> > > > > > > > The current
PRs suggest that operation-specific > > >>>> multi-object > > >>>> > > >
methods > > >>>> > > > > > can > > >>>> > > > > > > > fix individual cases,
while leaving the same contract > > >>>> question to > > >>>> > > > > recur
> > >>>> > > > > > > for > > >>>> > > > > > > > each new case. > > >>>> > >
> > > > > > > >>>> > > > > > > > I would also avoid defining a logical
change set as a > > >>>> database > > >>>> > > > > > > transaction > > >>>>
> > > > > > > around an entire REST request. > > >>>> > > > > > > > That
would tie the contract to the backend and > database, > > >>>> and > > >>>>
> > could > > >>>> > > > > keep > > >>>> > > > > > > the > > >>>> > > > > >
> > durable attempt open across slow external work. > > >>>> > > > > > > >
> > >>>> > > > > > > > So is the choice really between the two current >
manager > > >>>> > > > > > implementations, > > >>>> > > > > > > or > >
>>>> > > > > > > > do we first need to revisit the boundary and >
guarantees > > >>>> exposed > > >>>> > > to > > >>>> > > > > > their > >
>>>> > > > > > > > callers? > > >>>> > > > > > > > > > >>>> > > > > > > >
Cheers, > > >>>> > > > > > > > Robert > > >>>> > > > > > > > > > >>>> > > >
> > > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri
Bourlatchkov < > > >>>> > > > > dimas@apache.org > > >>>> > > > > > > > >
>>>> > > > > > > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > >
Hi Yufei, > > >>>> > > > > > > > > > > >>>> > > > > > > > > Thanks for the
link. I stand corrected. The "one > > atomic > > >>>> change > > >>>> > > >
per > > >>>> > > > > > > method" > > >>>> > > > > > > > > contract as
defined in javadoc does apply only to > > >>>> > > BasePersistence > > >>>>
> > > > > > > > and AtomicOperationMetaStoreManager (which delegates > > to
> > >>>> > > > > > > BasePersistence). > > >>>> > > > > > > > > > > >>>> >
> > > > > > > Note that all non-test Persistence implementations > in > >
>>>> the > > >>>> > > Polaris > > >>>> > > > > > > codebase > > >>>> > > >
> > > > > extend those classes (which is probably why I was > > >>>>
confused > > >>>> > > about > > >>>> > > > > > > atomicity > > >>>> > > > >
> > > > expectations). > > >>>> > > > > > > > > > > >>>> > > > > > > > >
However, this creates a gap in the Persistence SPI > > >>>> > >
specification. > > >>>> > > > > If > > >>>> > > > > > > > > other
PolarisMetaStoreManager implementations do not > > >>>> have to > > >>>> >
> > comply > > >>>> > > > > > > with > > >>>> > > > > > > > > this
principle, it will create a conceptual > difficulty > > >>>> at call > >
>>>> > > > > sites. > > >>>> > > > > > > How > > >>>> > > > > > > > > can
PolarisMetaStoreManager callers reason about > > >>>> consistency > > >>>>
> > and > > >>>> > > > > > > durability > > >>>> > > > > > > > > behaviours
in general? > > >>>> > > > > > > > > > > >>>> > > > > > > > > I believe we
need to address that as part of this > > >>>> discussion. > > >>>> > > > >
> > > > > > >>>> > > > > > > > > Cheers, > > >>>> > > > > > > > > Dmitri. >
> >>>> > > > > > > > > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at
9:29 PM Yufei Gu < > > >>>> flyrain000@gmail.com > > >>>> > > > > > >>>> >
> > > > wrote: > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > The
MetaStore SPI is currently defined with the > > >>>> idea that > > >>>> > >
one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>> > >
> > > > > > > means one atomic change. > > >>>> > > > > > > > > > > > >>>>
> > > > > > > > > If the "MetaStore SPI" refers to the interface > > >>>> >
> > > > > PolarisMetaStoreManager, I > > >>>> > > > > > > > > > don't think
we've ever state each method to be > > >>>> atomic. We did > > >>>> > > > >
> clarify > > >>>> > > > > > > > > > atomicity[1] in the interface
BasePersistence > > though. > > >>>> > > > > > > > > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > > > > 1. > > >>>> > > > > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> > > >
> > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > >
https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > > > > Yufei > > >>>> > > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at
11:27 AM Dmitri > > Bourlatchkov < > > >>>> > > > > > > dimas@apache.org> >
> >>>> > > > > > > > > > wrote: > > >>>> > > > > > > > > > > > >>>> > > > >
> > > > > > Hi Yufei, > > >>>> > > > > > > > > > > > > >>>> > > > > > > > >
> > I agree that JDBC transactions must be handled > > more > > >>>> > > >
explicitly. > > >>>> > > > > > > > > However, > > >>>> > > > > > > > > > >
I'm not sure that simply moving to > > >>>> > > > > > >
TransactionalMetaStoreManagerImpl is > > >>>> > > > > > > > > > >
sufficient. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > The
MetaStore SPI is currently defined with the > > >>>> idea that > > >>>> > >
one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>> > >
> > > > > > > > means one atomic change [1]. The "transactional" > > >>>>
MetaStore > > >>>> > > > > impl. > > >>>> > > > > > is > > >>>> > > > > > >
> > but a > > >>>> > > > > > > > > > > sub-case of that. It cannot alter
the high-level > > >>>> contract. > > >>>> > > > > > > > > > > > > >>>> > >
> > > > > > > > We could add SPI methods having multiple object > > >>>>
parameters > > >>>> > > to > > >>>> > > > > > > represent > > >>>> > > > >
> > > > > > grouped changes, but I am not sure it will be a > > >>>> sound
> > >>>> > > design. > > >>>> > > > > > This > > >>>> > > > > > > will > >
>>>> > > > > > > > > > > bloat the interface surfaces and require extra > >
>>>> impl. effort > > >>>> > > > for > > >>>> > > > > > > each > > >>>> > >
> > > > > > > > backend type. More importantly, adding multi-arg > > >>>>
change > > >>>> > > > methods > > >>>> > > > > > > still > > >>>> > > > > >
> > > > won't > > >>>> > > > > > > > > > > address the problem of reads
being consistent > with > > >>>> writes, > > >>>> > > > > > because > >
>>>> > > > > > > each > > >>>> > > > > > > > > > > method call will still
be independent regarding > > the > > >>>> data > > >>>> > > > stored > >
>>>> > > > > > in > > >>>> > > > > > > the > > >>>> > > > > > > > > > >
database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > I tend
to think we need to introduce a "change > > set" > > >>>> or > > >>>> > > >
"atomic > > >>>> > > > > > > batch" > > >>>> > > > > > > > > > > concept to
core Persistence and associate each > > REST > > >>>> API > > >>>> > > >
request > > >>>> > > > > > > with > > >>>> > > > > > > > > one > > >>>> > >
> > > > > > > > such change set, which will be committed (or > > rolled > >
>>>> back) > > >>>> > > at > > >>>> > > > > the > > >>>> > > > > > > end of
> > >>>> > > > > > > > > > the > > >>>> > > > > > > > > > > request. I
believe Ayush mentioned a similar > > >>>> concept in PR > > >>>> > > >
4939 > > >>>> > > > > > > [2]. In > > >>>> > > > > > > > > > > JDBC each
change set will naturally be > associated > > >>>> with an > > >>>> > > >
RDBMS > > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each
atomic > > >>>> change set > > >>>> > > > will > > >>>> > > > > be > > >>>>
> > > > > > > > > > associated with one CAS operation on the > > underlying
> > >>>> > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > >
> > > [1] > > >>>> > > > > > > > > >>>>
https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > > >>>> >
> > > > > > > > > > > >>>> > > > > > > > > > > [2] > > >>>> > > > > > > > >
>>>> https://github.com/apache/polaris/pull/4939#discussion_r3575719158 > >
>>>> > > > > > > > > > > > > >>>> > > > > > > > > > > Cheers, > > >>>> > >
> > > > > > > > Dmitri. > > >>>> > > > > > > > > > > > > >>>> > > > > > > >
> > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < > > >>>> > > >
flyrain000@gmail.com > > >>>> > > > > > > > >>>> > > > > > > wrote: > >
>>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > Thanks for raising
this, Dmitri. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > >
These are valid concerns, and they were > already > > >>>> > > recognized >
> >>>> > > > > when > > >>>> > > > > > > we > > >>>> > > > > > > > > > > >
introduced JDBC persistence to Polaris. At > that > > >>>> time, we > >
>>>> > > > > chose > > >>>> > > > > > > to use > > >>>> > > > > > > > > > >
> AtomicOperationMetaStoreManager for the JDBC > due > > >>>> to the > >
>>>> > > > > > > simplicity. I > > >>>> > > > > > > > > > > think > > >>>>
> > > > > > > > > > > most of the issues mentioned here can already > be >
> >>>> > > addressed > > >>>> > > > by > > >>>> > > > > > > > > > > >
TransactionalMetaStoreManagerImpl. > > >>>> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > For example, rename is already wrapped in a > > >>>>
transaction > > >>>> > > in > > >>>> > > > > > > > > > > >
TransactionalMetaStoreManagerImpl [1]. > > Similarly, > > >>>> catalog > >
>>>> > > > > > > creation, > > >>>> > > > > > > > > > which > > >>>> > > >
> > > > > > > > involves reading and creating multiple > objects, > > >>>>
is also > > >>>> > > > > > executed > > >>>> > > > > > > > > > within a > >
>>>> > > > > > > > > > > > transaction [2]. > > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > I see two possible directions: > > >>>> > >
> > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > >
> > > > > > >>>> > > > > > > > > > > > Modify
AtomicOperationMetaStoreManager > > >>>> together with > > >>>> > > the > >
>>>> > > > > > > > > persistence > > >>>> > > > > > > > > > > > backends
(such as JDBC) to provide the > > required > > >>>> > > > > consistency > >
>>>> > > > > > > > > > guarantees > > >>>> > > > > > > > > > > > for > >
>>>> > > > > > > > > > > > specific operations, similar to what > > >>>> >
> > > > > > > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > > >
> > > does. > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > > > > Migrate the persistence backends (such as
> > >>>> JDBC) to use > > >>>> > > > > > > > > > > >
TransactionalMetaStoreManagerImpl directly. > > We > > >>>> may > > >>>> >
> have > > >>>> > > > to > > >>>> > > > > > > deal > > >>>> > > > > > > > >
with > > >>>> > > > > > > > > > > > transactional semantic mismatches
across > > >>>> different > > >>>> > > > > > persistence > > >>>> > > > > >
> > > > > backends. > > >>>> > > > > > > > > > > > For example, we would
likely avoid using > > JDBC's > > >>>> > > > > > > > > >
`runWithinTransaction` > > >>>> > > > > > > > > > > > for > > >>>> > > > >
> > > > > > > single row updates, which adds additional > > >>>> overhead
and > > >>>> > > > > > > complexity > > >>>> > > > > > > > > > > > without
> > >>>> > > > > > > > > > > > benefits. > > >>>> > > > > > > > > > > > > >
>>>> > > > > > > > > > > > References: > > >>>> > > > > > > > > > > > > >
>>>> > > > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > > >>>> > > >
> > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > >
> > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> >
> > > > >>>> > > > > >>>> > > >
https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > > > > > >>>> >
> > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > >
> > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > >
> > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > >
>>>> > > > > > >>>> > > > > >>>> > > >
https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > Yufei > >
>>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > >
> > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri > > >>>> Bourlatchkov
< > > >>>> > > > > > > > > dimas@apache.org> > > >>>> > > > > > > > > > > >
wrote: > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Hi
all, > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > >
Ayush and Prithvi recently contributed a > > couple > > >>>> of > > >>>> >
> > > > interesting > > >>>> > > > > > > PRs: > > >>>> > > > > > > > > > >
> > [4939], [5035]. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > >
> > > > > It looks like people are starting to > encounter > > >>>> > > >
consistency > > >>>> > > > > > > issues > > >>>> > > > > > > > > in > >
>>>> > > > > > > > > > > > > JDBC persistence. > > >>>> > > > > > > > > > >
> > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight into
the > > >>>> underlying > > >>>> > > > > issues. > > >>>> > > > > > > They
> > >>>> > > > > > > > > > offer > > >>>> > > > > > > > > > > > >
incremental fixes that can work. However, I > > >>>> believe it > > >>>> >
> is > > >>>> > > > > > time > > >>>> > > > > > > for > > >>>> > > > > > >
> > > the > > >>>> > > > > > > > > > > > > Polaris community to review and
improve this > > >>>> area of > > >>>> > > the > > >>>> > > > > > >
codebase > > >>>> > > > > > > > > > > > > holistically. > > >>>> > > > > >
> > > > > > > > > >>>> > > > > > > > > > > > > By this, I mean finding a
solution that can > be > > >>>> applied > > >>>> > > to > > >>>> > > > >
all > > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC,
> NoSQL) > > >>>> and > > >>>> > > > addresses > > >>>> > > > > > > these >
> >>>> > > > > > > > > > > > > aspects: > > >>>> > > > > > > > > > > > > >
> >>>> > > > > > > > > > > > > * Supporting concurrent and consistent >
changes > > >>>> where > > >>>> > > the > > >>>> > > > > > > service > >
>>>> > > > > > > > > > reads > > >>>> > > > > > > > > > > > > and validates
current catalog state, then > > >>>> commits a > > >>>> > > > > change > >
>>>> > > > > > > (e.g. > > >>>> > > > > > > > > > > > > name clashes during
renames). > > >>>> > > > > > > > > > > > > * Supporting consistent but
independent > > changes > > >>>> to RBAC > > >>>> > > > > > grants > > >>>>
> > > > > > and > > >>>> > > > > > > > > > > > > MetaStore entities. This
independence is > > >>>> needed to > > >>>> > > > > support > > >>>> > > >
> > > > > > > > > external authorizers like OPA and Ranger. > > >>>> > > >
> > > > > > > > > * Supporting atomic changes across multiple > > >>>>
similar > > >>>> > > > > entities. > > >>>> > > > > > > > > > > > > *
Supporting authorization-based filtering > of > > >>>> list > > >>>> > > >
> operations > > >>>> > > > > > > (cf. > > >>>> > > > > > > > > > > > >
[4831]). > > >>>> > > > > > > > > > > > > * Supporting credential-vending
decisions > that > > >>>> are > > >>>> > > rooted > > >>>> > > > > in > >
>>>> > > > > > > the > > >>>> > > > > > > > > > > > > exact state of the
catalog. > > >>>> > > > > > > > > > > > > * Supporting server-side retries
for > transient > > >>>> > > > persistence > > >>>> > > > > > > failures >
> >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures). >
> >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Please
share your comments and ideas. > > >>>> > > > > > > > > > > > > > > >>>> >
> > > > > > > > > > > [4831] > > >>>>
https://github.com/apache/polaris/pull/4831 > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > [4939] > > >>>>
https://github.com/apache/polaris/pull/4939 > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > [5035] > > >>>>
https://github.com/apache/polaris/pull/5035 > > >>>> > > > > > > > > > > >
> > > >>>> > > > > > > > > > > > > Thanks, > > >>>> > > > > > > > > > > > >
Dmitri > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > >
> > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > >
> >>>> > > > > >>>> > > >>> > > >
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi EJ,
> 1. Which backend assumptions should we design for? [...]
I know I used the term "transaction" a lot during the call, but the real
challenge is about establishing an SPI with clear contracts that could be
implemented by JDBC and NoSQL persistence using the database features
natural to each model.
I am fairly confident that a JDBC impl. is possible with Serializable
transaction. We should be able to solve all the problems raised in this
thread. However, that will make in-memory caching impossible because the
database needs to see all the reads (and writes) in a Tx to be able to
ensure serializable guarantees. If a cache prevents a read from hitting the
database, that read (e.g. storage config) will not be a factor in
serializability.
This connects to my earlier question about the in-memory cache and JDBC
(title: Usefulness of InMemoryEntityCache)
Are different approaches possible with JDBC? Probably. What is involved?
This depends on the SPI contract, I think.
> 2. How do we protect the data used by validation? [...]
The point mentioned by Robert previously IIRC, is that we need to ground
all runtime decisions to a particular state of the catalog.
If at commit time the state of the catalog in the database shifts from what
was used during validation, the request has to restart from scratch.
Current JDBC Persistence does this only for relationships between entities
and grants and between versions of the same entity.
Relationships between table locations and storage config are not actualized
for the purpose of state consistency tracking.
From my POV we need to begin by formulating this Persistence state in the
Persistence SPI terms. Then, think how we can implement that for JDBC.
NoSQL Persistence already has a per-catalog state (not granular), so wiring
the new SPI to NoSQL should not be too hard, I hope.
The state in the SPI does not have to be global to the catalog (as in
NoSQL), but it needs to be able to express the idea of which elements were
engaged during request processing so serve as input to Persistence impl.
for cosistency guarantees. If Persistence caches data in memory (for
example), it will have to do so in a way that does not violate consistency
expectations.
From my POV we could begin by defining a "Persistence Session" concept.
Each request is 1:1 with a session. All reads and writes go through the
session. The next step would be the JDBC impl., which probably needs some
brainstorming... but I wonder if we could reach a consensus at a high level
first.
WDYT?
Thanks,
Dmitri.
On Tue, Sep 22, 2026 at 5:55 PM EJ Wang <ej...@gmail.com>
wrote:
> Thanks for the recap, Dmitri. I'd like to follow up on the three points I
> raised during the sync. We didn't have much time to discuss them, so it
> would help to confirm these before choosing an SPI design.
>
> 1. Which backend assumptions should we design for?
>
> Within one Polaris server instance, do we assume entities, grants and
> secrets all use the same database backend, or must we support mixing
> backends across them?
> Must every supported persistence implementation provide transactional
> guarantees? Does the database itself need to provide transactions, or can
> Polaris build that capability on top of simpler database operations?
>
> I'm happy to work with a homogeneous assumption if that's the intended
> scope. Existing NoSQL illustrates the second question: it can publish
> changes to multiple objects under one reference using a single conditional
> pointer update.
>
> 2. How do we protect the data used by validation?
>
> Your storage-configuration and location-overlap examples make the problem
> concrete. Consider these races:
>
> A table creation passes validation against the catalog's allowed locations,
> then another request changes that configuration before the table is
> committed.
> Two table-creation requests each check for overlapping locations, both see
> no overlap, and then create tables at conflicting locations.
>
> Checking only the record being written cannot cover these dependencies. The
> first also depends on the catalog configuration. The second depends on the
> absence of another overlapping table.
>
> Could the persistence layer protect these reads together with the writes,
> using backend-internal versions/tokens or database isolation? Shared code
> would still need to identify the required reads and checks. Could we keep
> the mechanism for protecting them inside each backend?
>
> 3. Business rules are duplicated across backend implementations
>
> Today, the TreeMap
> <
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481
> >
> and JDBC
> <
> https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046
> >
> implementations each load a secret, check its principal, apply the
> rotation/reset steps and write it back.
>
> Changing these business rules therefore requires keeping multiple
> implementations in sync. A fix applied to one backend can be missed in
> another, causing the same Polaris operation to behave differently depending
> on the database.
>
> My proposal is to implement each business workflow once, against a common
> set of storage operations with defined consistency and failure semantics.
> The shared workflow should not need to know whether those operations are
> implemented through JDBC transactions, NoSQL reference CAS or another
> mechanism. Each backend would implement the same storage contract, keeping
> its execution details internal. A new database backend could then reuse the
> existing business workflows.
>
> Do these seem like reasonable starting points? We can then evaluate the SPI
> against the original failure cases on both JDBC and existing NoSQL.
>
> -ej
>
> On Fri, Sep 18, 2026 at 12:10 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Side note: I just came across [5541], which shows a case where a failed
> > request leaves persisted side effects.
> >
> > This can be seen as a coding mistake, of course. However, this is about
> > executing one request as a single, atomic unit. This is something that
> > probably needs foundational support and coding patterns in Polaris code.
> >
> > Cheers,
> > Dmitri.
> > [5541] https://github.com/apache/polaris/pull/5541
> >
> > On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi All,
> > >
> > > Here's a recap of the Community Sync discussion as I remember it. I'm
> > sure
> > > my recollection is not complete, so please add your thoughts to this
> > thread.
> > >
> > > * Polaris traditionally relies on entity version number to ensure
> > > consistency of changes.
> > >
> > > This works well for internal RBAC grants, which also have version
> numbers
> > > cross-referenced at the Persistence layer.
> > >
> > > I do not think this works in more general cases like validating
> locations
> > > wrt catalog-level storage configuration.
> > >
> > > Another difficult case involves location overlaps between concurrent
> > table
> > > creations.
> > >
> > > * We talked about how Persistence calls related to RDBMS transactions
> (in
> > > the JDBC persistence case).
> > >
> > > There were some concerns raised about whether we need to bind
> Persistence
> > > API to transactions explicitly.
> > >
> > > I think it is a valid concern. At the same time, the Persistence API
> > > should be clear about consistency guarantees across all backend
> > > implementations. I think this needs more attention now that the matter
> of
> > > milti-entioty changes became prominent in [5035].
> > >
> > > I personally believe that RDBMS-based Persistence in Polaris has to use
> > > Serializable transaction isolation. Consequently, the core code needs
> to
> > > provide mechanisms to allow the Persistence implementation to properly
> > > connect reads and writes from API requests to JDBC transactions...
> > whether
> > > we use "transation" as an explicit terms in the SPI or not.
> > >
> > > Note that currently, the same API request performs reads and writes in
> > > _separate_ RDBMS transactions (even separate connections).
> > >
> > > * We did not discuss Persistence SPI implications in the call, but
> > > currently each Persistence call is assumed to be one distinct and
> atomic
> > > change (please correct me if I'm wrong).
> > >
> > > Consequently, supporting new use cases involves introducing new SPI
> > > methods with many parameters. This can be seen in PR [5035]. This tends
> > to
> > > overcomplicate the SPI.
> > >
> > > Ideally, I think the SPI should be simplified to deal with multi-entity
> > > and single-entity changes in a coherent manner to avoid any ambiguity
> > about
> > > which method the caller should use.
> > >
> > > [5035] https://github.com/apache/polaris/pull/5035
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > wrote:
> > >
> > >> Heads up: This discussion is on the Community Sync call agenda for
> Sept
> > >> 17.
> > >>
> > >> Interested parties, please plan to attend, if possible :)
> > >>
> > >> Cheers,
> > >> Dmitri.
> > >>
> > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <di...@apache.org>
> > >> wrote:
> > >>
> > >>> Hi JB,
> > >>>
> > >>> I've added an agenda item to the next community sync call for this
> > (Sept
> > >>> 17, 2026).
> > >>>
> > >>> Re: key points: did you mean a real GH discussion or dev email (this
> > >>> thread)? Just double checking :) I'm fine with either approach.
> > >>>
> > >>> Cheers,
> > >>> Dmitri.
> > >>>
> > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > >>> wrote:
> > >>>
> > >>>> Hi Dmitri,
> > >>>>
> > >>>> I agree on the need to align on these points, though I'm not
> entirely
> > >>>> sure a dedicated meeting is necessary. Let's start by using some
> time
> > >>>> during the next community meeting to discuss it.
> > >>>>
> > >>>> In the meantime, what if we outline the key points in a GitHub
> > >>>> Discussion first? We can then use that as a reference during the
> > >>>> meeting.
> > >>>>
> > >>>> Regards,
> > >>>> JB
> > >>>>
> > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <
> dimas@apache.org
> > >
> > >>>> wrote:
> > >>>> >
> > >>>> > Hi All,
> > >>>> >
> > >>>> > My impression from this thread is that it might be time for a call
> > to
> > >>>> > discuss all the related issues and try to arrive at a shared
> > >>>> implementation
> > >>>> > plan.
> > >>>> >
> > >>>> > I know meetings are not ideal, but in this case it might be
> > >>>> beneficial as a
> > >>>> > means for achieving a common understanding of the set of problems
> > and
> > >>>> > priorities related to this thread. A dedicated metrics meeting
> > worked
> > >>>> well
> > >>>> > from my POV.
> > >>>> >
> > >>>> > Allocating a time slot in the next community meeting might be an
> > >>>> option,
> > >>>> > although I think we might need the full hour in this case.
> > >>>> >
> > >>>> > WDYT?
> > >>>> >
> > >>>> > Thanks,
> > >>>> > Dmitri.
> > >>>> >
> > >>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <
> > >>>> prithvisivasankar@gmail.com>
> > >>>> > wrote:
> > >>>> >
> > >>>> > > Hi all,
> > >>>> > >
> > >>>> > > I rewrote https://github.com/apache/polaris/pull/5035,
> following
> > >>>> the
> > >>>> > > discussion to focus only on the SPI foundation.
> > >>>> > >
> > >>>> > > What is in the PR now:
> > >>>> > >
> > >>>> > > - A written manager-level consistency contract in
> > >>>> > > site/content/in-dev/unreleased/persistence-consistency.md.
> > >>>> > > - CAS-aware EntityMutation / GrantMutation records.
> > >>>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet,
> with
> > >>>> backends
> > >>>> > > opting in via supportsAtomicMixedCommit().
> > >>>> > > - commitChangeSet implementations for the in-memory TreeMap
> > >>>> backend and
> > >>>> > > JDBC.
> > >>>> > > - AtomicOperationMetaStoreManager and
> > >>>> TransactionalMetaStoreManagerImpl
> > >>>> > > grant/revoke paths now build one change set per operation,
> > >>>> falling back
> > >>>> > > to
> > >>>> > > individual operations when the backend does not support mixed
> > >>>> commits.
> > >>>> > > - Tests for the change-set API and TreeMap atomic commits.
> > >>>> > >
> > >>>> > > I intentionally left out:
> > >>>> > >
> > >>>> > > - Entity deletes in MetaStoreChangeSet.
> > >>>> > > - Refactoring createCatalog, dropEntity, and renameEntity to
> > use
> > >>>> change
> > >>>> > > sets.
> > >>>> > > - NoSQL support for commitChangeSet.
> > >>>> > >
> > >>>> > > I am treating this as Phase 1 so the contract and the SPI
> > primitive
> > >>>> can be
> > >>>> > > reviewed before the larger operation migrations land.
> > >>>> > >
> > >>>> > > Could you please take a look? In particular, I would like to
> know
> > >>>> whether
> > >>>> > > the contract captures the consensus so far, or if it needs to
> > >>>> address retry
> > >>>> > > / external-work coordination before we merge this foundation.
> > >>>> > >
> > >>>> > > Thanks,
> > >>>> > > Prithvi S
> > >>>> > >
> > >>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <
> > >>>> dimas@apache.org>
> > >>>> > > wrote:
> > >>>> > >
> > >>>> > > > Hi Robert,
> > >>>> > > >
> > >>>> > > > I agree that the client-visible operation (e.g. REST API
> > request)
> > >>>> > > consists
> > >>>> > > > of many distinct phases. The database / persistence changes
> are
> > >>>> just one
> > >>>> > > of
> > >>>> > > > these phases. STS (as an example) is another. My point about
> > >>>> transactions
> > >>>> > > > in JDBC Persistence applies to the former (database changes)
> > >>>> phase - one
> > >>>> > > > retry attempt there should ideally involve exactly one
> > >>>> transaction.
> > >>>> > > >
> > >>>> > > > We certainly need logic inside Polaris Servers to to
> coordinate
> > >>>> requests
> > >>>> > > to
> > >>>> > > > various external systems depending on outcomes from previous
> > >>>> request
> > >>>> > > > processing phases.
> > >>>> > > >
> > >>>> > > > I also agree that validation based on database state should be
> > >>>> repeated
> > >>>> > > > from scratch if we retry the database changes.
> > >>>> > > >
> > >>>> > > > However, internal RBAC authorization naturally has to happen
> > >>>> inside the
> > >>>> > > > database update phase (read-check-write). IIRC, that is part
> of
> > >>>> the
> > >>>> > > > implicit consistency guarantees, which were discussed during
> > early
> > >>>> > > project
> > >>>> > > > months (but never got written down in full clarity,
> > >>>> unfortunately). If
> > >>>> > > the
> > >>>> > > > Authorization check is inside that phase for internal RBAC, I
> > >>>> suppose it
> > >>>> > > > will be there for other Authorizers too and that can involve
> > >>>> external
> > >>>> > > > systems (e.g. OPA / Ranger). We can certainly expect
> > >>>> authorization to be
> > >>>> > > > efficient, but I'm not sure how quick it can be in practice.
> > >>>> There's
> > >>>> > > > certainly potential for delays in some (perhaps infrequent)
> > cases.
> > >>>> > > >
> > >>>> > > > Cheers,
> > >>>> > > > Dmitri.
> > >>>> > > >
> > >>>> > > >
> > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <sn...@snazy.de>
> > >>>> wrote:
> > >>>> > > >
> > >>>> > > > > Hi Dmitri,
> > >>>> > > > >
> > >>>> > > > > I agree that one logical operation needs a consistent view
> of
> > >>>> the whole
> > >>>> > > > > backend state.
> > >>>> > > > > What worries me is treating one long JDBC transaction as the
> > >>>> boundary
> > >>>> > > of
> > >>>> > > > > that operation and then retrying the complete request if the
> > >>>> commit
> > >>>> > > > fails.
> > >>>> > > > >
> > >>>> > > > > The request can also write metadata to object storage or
> call
> > >>>> STS.
> > >>>> > > > > Each system can leave us with an outcome whose certainty we
> > >>>> cannot
> > >>>> > > know.
> > >>>> > > > > The database may have committed before the connection was
> > lost,
> > >>>> an
> > >>>> > > > > object-store write may have succeeded before a timeout, or
> STS
> > >>>> may have
> > >>>> > > > > issued credentials before its response was lost.
> > >>>> > > > > A database rollback cannot undo any of those effects, and
> > >>>> retrying the
> > >>>> > > > > whole request may repeat them.
> > >>>> > > > >
> > >>>> > > > > So I think the contract has to separate the overall
> operation
> > >>>> from each
> > >>>> > > > > attempt to commit it.
> > >>>> > > > > Each database or backend commit attempt should stay short.
> > >>>> > > > > When we retry, validation and authorization need to use the
> > >>>> state for
> > >>>> > > > that
> > >>>> > > > > new attempt, and we need clear rules for which external work
> > >>>> can be
> > >>>> > > > reused,
> > >>>> > > > > repeated, cleaned up, or reconciled after an uncertain
> > outcome.
> > >>>> > > > >
> > >>>> > > > > Some contextual state could be useful for carrying that
> state.
> > >>>> > > > > But the JDBC connection and transaction should be an
> > >>>> implementation
> > >>>> > > > detail
> > >>>> > > > > inside it, not the boundary of the complete operation.
> > >>>> > > > >
> > >>>> > > > > Cheers,
> > >>>> > > > > Robert
> > >>>> > > > >
> > >>>> > > > >
> > >>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <
> > >>>> dimas@apache.org
> > >>>> > > >
> > >>>> > > > > wrote:
> > >>>> > > > >
> > >>>> > > > > > Hi All,
> > >>>> > > > > >
> > >>>> > > > > > The idea of a backend-agnostic change-set primitive sounds
> > >>>> good.
> > >>>> > > > > >
> > >>>> > > > > > However, I am not sure it is sufficient for all the use
> > cases
> > >>>> we
> > >>>> > > > touched
> > >>>> > > > > > here. More specifically, the state read by validation code
> > is
> > >>>> not
> > >>>> > > > > > necessarily reflected in the change set:
> > >>>> > > > > >
> > >>>> > > > > > * Unchanged entities might be considered by validation,
> but
> > >>>> changed
> > >>>> > > in
> > >>>> > > > a
> > >>>> > > > > > parallel request.
> > >>>> > > > > > * Even for changed entities the state read by validation
> may
> > >>>> differ
> > >>>> > > > from
> > >>>> > > > > > the state being altered. Some validation code talks to the
> > >>>> MetaStore
> > >>>> > > > > > directly, outside of the data produced by the Resolver.
> > >>>> > > > > >
> > >>>> > > > > > I do not think Polaris offers any explicit mechanisms
> (ATM)
> > >>>> to ensure
> > >>>> > > > > > consistency between these reads and subsequent writes.
> > >>>> > > > > >
> > >>>> > > > > > As far as JDBC goes, running an overarching Transaction
> > >>>> across all
> > >>>> > > > reads
> > >>>> > > > > > and writes in the same request, at the SERIALIZABLE
> > isolation
> > >>>> level
> > >>>> > > in
> > >>>> > > > > the
> > >>>> > > > > > backing RDBMS could solve the problem, I think.
> > >>>> > > > > >
> > >>>> > > > > > Indeed, such a transaction might be long to accommodate
> > calls
> > >>>> made by
> > >>>> > > > > > Polaris to external storage, etc. However, is that a
> > problem?
> > >>>> I think
> > >>>> > > > the
> > >>>> > > > > > alternative is for the JDBC Persistence impl. to
> "manually"
> > >>>> track all
> > >>>> > > > > reads
> > >>>> > > > > > under the same request and redo them in the small
> > transaction
> > >>>> that
> > >>>> > > > > persists
> > >>>> > > > > > the writes. This will also add RDBMS overhead and require
> > >>>> complex
> > >>>> > > code
> > >>>> > > > in
> > >>>> > > > > > Polaris to handle the data properly.
> > >>>> > > > > >
> > >>>> > > > > > Tracking the request-wide transaction can be done only for
> > >>>> JDBC
> > >>>> > > without
> > >>>> > > > > > leaking "transaction" concepts to the NoSQL Persistence, I
> > >>>> think.
> > >>>> > > NoSQL
> > >>>> > > > > > will use other mechanisms to ensure read/write
> consistency.
> > >>>> > > > > >
> > >>>> > > > > > Connecting to Robert's email (a parallel branch in this
> > >>>> discussion),
> > >>>> > > > I'd
> > >>>> > > > > > like to propose this approach:
> > >>>> > > > > >
> > >>>> > > > > > * Each request establishes a "Data Context"
> > >>>> > > > > > - In JDBC the Data Context corresponds to a JDBC
> > Connection
> > >>>> + Tx
> > >>>> > > > > > - In NoSQL the Data Context tracks one or more reference
> > >>>> hashes
> > >>>> > > > > > * All Persistence access in the same request goes through
> > the
> > >>>> same
> > >>>> > > Data
> > >>>> > > > > > Context
> > >>>> > > > > > * All Persistence changes are committed once at the end of
> > the
> > >>>> > > request
> > >>>> > > > > > - Not all changes have to be globally atomic. For
> example,
> > >>>> changes
> > >>>> > > in
> > >>>> > > > > > Catalogs A and B do not have to be atomic with respect to
> > >>>> each other.
> > >>>> > > > We
> > >>>> > > > > > can go deeper into this later. This is relevant to NoSQL.
> > >>>> > > > > > - Transactional backends like JDBC can, of course,
> choose
> > >>>> to make
> > >>>> > > all
> > >>>> > > > > > changes globally atomic.
> > >>>> > > > > > * A commit can fail in two main ways:
> > >>>> > > > > > - A retriable failure like an optimistic lock error or
> Tx
> > >>>> > > > > serializability
> > >>>> > > > > > error
> > >>>> > > > > > - A non-triable logical error (e.g. entity not found)
> > >>>> > > > > > * On a retriable error the whole request is re-attempted
> (as
> > >>>> if
> > >>>> > > > > resubmitted
> > >>>> > > > > > by a client) a few times (configurable timeout).
> > >>>> > > > > >
> > >>>> > > > > > WDYT?
> > >>>> > > > > >
> > >>>> > > > > > Cheers,
> > >>>> > > > > > Dmitri.
> > >>>> > > > > >
> > >>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré <
> > >>>> > > jb@nanthrax.net
> > >>>> > > > >
> > >>>> > > > > > wrote:
> > >>>> > > > > >
> > >>>> > > > > > > Hi all
> > >>>> > > > > > >
> > >>>> > > > > > > I think Robert has a good point.
> > >>>> > > > > > >
> > >>>> > > > > > > If the atomicity guarantee lives only in
> BasePersistence,
> > >>>> then the
> > >>>> > > > > > > manager contract can't tell a caller whether the state
> it
> > >>>> read for
> > >>>> > > > > > > validation/authorization/credential-vending is the same
> > >>>> state that
> > >>>> > > > > > > eventually commits. That's the actual bug class behind
> the
> > >>>> JDBC
> > >>>> > > > > > > symptoms (not any single operation being non-atomic, but
> > the
> > >>>> > > contract
> > >>>> > > > > > > being silent about it. Every operation-specific fix
> (like
> > >>>> #4939,
> > >>>> > > > #5035
> > >>>> > > > > > > or #5095) re-answers this question locally and it keeps
> > >>>> recurring.
> > >>>> > > So
> > >>>> > > > > > > the deliverable should start with a written consistency
> > >>>> contract at
> > >>>> > > > > > > the manager level.
> > >>>> > > > > > >
> > >>>> > > > > > > I'm not sure migrating everyone to
> > >>>> > > TransactionalMetaStoreManagerImpl
> > >>>> > > > > > > is a good idea. It would tie the logical change set to a
> > DB
> > >>>> > > > > > > transaction spanning the REST request. It means:
> > >>>> > > > > > > - it holds a durable transaction open across slow
> external
> > >>>> work
> > >>>> > > > > > > (credential vending, OPA, Ranger, ...)
> > >>>> > > > > > > - it doesn't map to NoSQL
> > >>>> > > > > > > - It wraps single-row updates in runWiithinTransaction,
> > >>>> which is a
> > >>>> > > > > > overhead
> > >>>> > > > > > >
> > >>>> > > > > > > So, I think the transactional manager isn't a portable
> > >>>> target. It's
> > >>>> > > > > > > "only" one backend's strategy.
> > >>>> > > > > > >
> > >>>> > > > > > > I think Privthi's approach is right. A backend-agnostic
> > >>>> change-set
> > >>>> > > > > > > primitive with a documented fallback is the correct
> shape.
> > >>>> There is
> > >>>> > > > > > > one caveat: without carrying the original entitiy (not
> > just
> > >>>> the new
> > >>>> > > > > > > one) the change set can't express optimistic
> concurrency.
> > >>>> > > > > > >
> > >>>> > > > > > > I propose the following multi-steps approach:
> > >>>> > > > > > > 1. We write the manager-level consistency contract as
> > >>>> Robert asked.
> > >>>> > > > It
> > >>>> > > > > > > should include the explicit statement that a logical
> > change
> > >>>> set is
> > >>>> > > > not
> > >>>> > > > > > > a request-scope DB transaction.
> > >>>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency
> > pattern)
> > >>>> > > baseline
> > >>>> > > > > > > first-class in EntityMutation before merging the SPI
> > >>>> > > > > > > 3. Then we refactor
> createCatalog/dropEntity/renameEntity.
> > >>>> > > > > > >
> > >>>> > > > > > > Thoughts?
> > >>>> > > > > > >
> > >>>> > > > > > > Regards
> > >>>> > > > > > > JB
> > >>>> > > > > > >
> > >>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <
> > >>>> snazy@snazy.de>
> > >>>> > > wrote:
> > >>>> > > > > > > >
> > >>>> > > > > > > > Hi all,
> > >>>> > > > > > > >
> > >>>> > > > > > > > Yufei's clarification about where the atomicity
> > guarantee
> > >>>> is
> > >>>> > > > defined
> > >>>> > > > > > > seems
> > >>>> > > > > > > > important.
> > >>>> > > > > > > > If it is a property of the lower-level BasePersistence
> > >>>> contract
> > >>>> > > > > rather
> > >>>> > > > > > > than
> > >>>> > > > > > > > the general PolarisMetaStoreManager contract, the
> > general
> > >>>> > > contract
> > >>>> > > > > does
> > >>>> > > > > > > not
> > >>>> > > > > > > > tell callers whether the state used for validation,
> > >>>> > > authorization,
> > >>>> > > > or
> > >>>> > > > > > > > credential vending is consistent with the change that
> > >>>> eventually
> > >>>> > > > > > commits.
> > >>>> > > > > > > >
> > >>>> > > > > > > > The current PRs suggest that operation-specific
> > >>>> multi-object
> > >>>> > > > methods
> > >>>> > > > > > can
> > >>>> > > > > > > > fix individual cases, while leaving the same contract
> > >>>> question to
> > >>>> > > > > recur
> > >>>> > > > > > > for
> > >>>> > > > > > > > each new case.
> > >>>> > > > > > > >
> > >>>> > > > > > > > I would also avoid defining a logical change set as a
> > >>>> database
> > >>>> > > > > > > transaction
> > >>>> > > > > > > > around an entire REST request.
> > >>>> > > > > > > > That would tie the contract to the backend and
> database,
> > >>>> and
> > >>>> > > could
> > >>>> > > > > keep
> > >>>> > > > > > > the
> > >>>> > > > > > > > durable attempt open across slow external work.
> > >>>> > > > > > > >
> > >>>> > > > > > > > So is the choice really between the two current
> manager
> > >>>> > > > > > implementations,
> > >>>> > > > > > > or
> > >>>> > > > > > > > do we first need to revisit the boundary and
> guarantees
> > >>>> exposed
> > >>>> > > to
> > >>>> > > > > > their
> > >>>> > > > > > > > callers?
> > >>>> > > > > > > >
> > >>>> > > > > > > > Cheers,
> > >>>> > > > > > > > Robert
> > >>>> > > > > > > >
> > >>>> > > > > > > >
> > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov <
> > >>>> > > > > dimas@apache.org
> > >>>> > > > > > >
> > >>>> > > > > > > > wrote:
> > >>>> > > > > > > >
> > >>>> > > > > > > > > Hi Yufei,
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > Thanks for the link. I stand corrected. The "one
> > atomic
> > >>>> change
> > >>>> > > > per
> > >>>> > > > > > > method"
> > >>>> > > > > > > > > contract as defined in javadoc does apply only to
> > >>>> > > BasePersistence
> > >>>> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates
> > to
> > >>>> > > > > > > BasePersistence).
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > Note that all non-test Persistence implementations
> in
> > >>>> the
> > >>>> > > Polaris
> > >>>> > > > > > > codebase
> > >>>> > > > > > > > > extend those classes (which is probably why I was
> > >>>> confused
> > >>>> > > about
> > >>>> > > > > > > atomicity
> > >>>> > > > > > > > > expectations).
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > However, this creates a gap in the Persistence SPI
> > >>>> > > specification.
> > >>>> > > > > If
> > >>>> > > > > > > > > other PolarisMetaStoreManager implementations do not
> > >>>> have to
> > >>>> > > > comply
> > >>>> > > > > > > with
> > >>>> > > > > > > > > this principle, it will create a conceptual
> difficulty
> > >>>> at call
> > >>>> > > > > sites.
> > >>>> > > > > > > How
> > >>>> > > > > > > > > can PolarisMetaStoreManager callers reason about
> > >>>> consistency
> > >>>> > > and
> > >>>> > > > > > > durability
> > >>>> > > > > > > > > behaviours in general?
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > I believe we need to address that as part of this
> > >>>> discussion.
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > Cheers,
> > >>>> > > > > > > > > Dmitri.
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <
> > >>>> flyrain000@gmail.com
> > >>>> > > >
> > >>>> > > > > > wrote:
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with the
> > >>>> idea that
> > >>>> > > one
> > >>>> > > > > > > method
> > >>>> > > > > > > > > call
> > >>>> > > > > > > > > > means one atomic change.
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > If the "MetaStore SPI" refers to the interface
> > >>>> > > > > > > PolarisMetaStoreManager, I
> > >>>> > > > > > > > > > don't think we've ever state each method to be
> > >>>> atomic. We did
> > >>>> > > > > > clarify
> > >>>> > > > > > > > > > atomicity[1] in the interface BasePersistence
> > though.
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > 1.
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > >
> > >>>> > > > > >
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>>
> >
> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > Yufei
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri
> > Bourlatchkov <
> > >>>> > > > > > > dimas@apache.org>
> > >>>> > > > > > > > > > wrote:
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > > Hi Yufei,
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > I agree that JDBC transactions must be handled
> > more
> > >>>> > > > explicitly.
> > >>>> > > > > > > > > However,
> > >>>> > > > > > > > > > > I'm not sure that simply moving to
> > >>>> > > > > > > TransactionalMetaStoreManagerImpl is
> > >>>> > > > > > > > > > > sufficient.
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with the
> > >>>> idea that
> > >>>> > > one
> > >>>> > > > > > > method
> > >>>> > > > > > > > > call
> > >>>> > > > > > > > > > > means one atomic change [1]. The "transactional"
> > >>>> MetaStore
> > >>>> > > > > impl.
> > >>>> > > > > > is
> > >>>> > > > > > > > > but a
> > >>>> > > > > > > > > > > sub-case of that. It cannot alter the high-level
> > >>>> contract.
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > We could add SPI methods having multiple object
> > >>>> parameters
> > >>>> > > to
> > >>>> > > > > > > represent
> > >>>> > > > > > > > > > > grouped changes, but I am not sure it will be a
> > >>>> sound
> > >>>> > > design.
> > >>>> > > > > > This
> > >>>> > > > > > > will
> > >>>> > > > > > > > > > > bloat the interface surfaces and require extra
> > >>>> impl. effort
> > >>>> > > > for
> > >>>> > > > > > > each
> > >>>> > > > > > > > > > > backend type. More importantly, adding multi-arg
> > >>>> change
> > >>>> > > > methods
> > >>>> > > > > > > still
> > >>>> > > > > > > > > > won't
> > >>>> > > > > > > > > > > address the problem of reads being consistent
> with
> > >>>> writes,
> > >>>> > > > > > because
> > >>>> > > > > > > each
> > >>>> > > > > > > > > > > method call will still be independent regarding
> > the
> > >>>> data
> > >>>> > > > stored
> > >>>> > > > > > in
> > >>>> > > > > > > the
> > >>>> > > > > > > > > > > database.
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > I tend to think we need to introduce a "change
> > set"
> > >>>> or
> > >>>> > > > "atomic
> > >>>> > > > > > > batch"
> > >>>> > > > > > > > > > > concept to core Persistence and associate each
> > REST
> > >>>> API
> > >>>> > > > request
> > >>>> > > > > > > with
> > >>>> > > > > > > > > one
> > >>>> > > > > > > > > > > such change set, which will be committed (or
> > rolled
> > >>>> back)
> > >>>> > > at
> > >>>> > > > > the
> > >>>> > > > > > > end of
> > >>>> > > > > > > > > > the
> > >>>> > > > > > > > > > > request. I believe Ayush mentioned a similar
> > >>>> concept in PR
> > >>>> > > > 4939
> > >>>> > > > > > > [2]. In
> > >>>> > > > > > > > > > > JDBC each change set will naturally be
> associated
> > >>>> with an
> > >>>> > > > RDBMS
> > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each atomic
> > >>>> change set
> > >>>> > > > will
> > >>>> > > > > be
> > >>>> > > > > > > > > > > associated with one CAS operation on the
> > underlying
> > >>>> > > database.
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > [1]
> > >>>> > > > > > >
> > >>>> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > [2]
> > >>>> > > > > > >
> > >>>> https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > Cheers,
> > >>>> > > > > > > > > > > Dmitri.
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
> > >>>> > > > flyrain000@gmail.com
> > >>>> > > > > >
> > >>>> > > > > > > wrote:
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > > > > Thanks for raising this, Dmitri.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > These are valid concerns, and they were
> already
> > >>>> > > recognized
> > >>>> > > > > when
> > >>>> > > > > > > we
> > >>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At
> that
> > >>>> time, we
> > >>>> > > > > chose
> > >>>> > > > > > > to use
> > >>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC
> due
> > >>>> to the
> > >>>> > > > > > > simplicity. I
> > >>>> > > > > > > > > > > think
> > >>>> > > > > > > > > > > > most of the issues mentioned here can already
> be
> > >>>> > > addressed
> > >>>> > > > by
> > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > For example, rename is already wrapped in a
> > >>>> transaction
> > >>>> > > in
> > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1].
> > Similarly,
> > >>>> catalog
> > >>>> > > > > > > creation,
> > >>>> > > > > > > > > > which
> > >>>> > > > > > > > > > > > involves reading and creating multiple
> objects,
> > >>>> is also
> > >>>> > > > > > executed
> > >>>> > > > > > > > > > within a
> > >>>> > > > > > > > > > > > transaction [2].
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > I see two possible directions:
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > 1.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager
> > >>>> together with
> > >>>> > > the
> > >>>> > > > > > > > > persistence
> > >>>> > > > > > > > > > > > backends (such as JDBC) to provide the
> > required
> > >>>> > > > > consistency
> > >>>> > > > > > > > > > guarantees
> > >>>> > > > > > > > > > > > for
> > >>>> > > > > > > > > > > > specific operations, similar to what
> > >>>> > > > > > > > > > TransactionalMetaStoreManagerImpl
> > >>>> > > > > > > > > > > > does.
> > >>>> > > > > > > > > > > > 2.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > Migrate the persistence backends (such as
> > >>>> JDBC) to use
> > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl directly.
> > We
> > >>>> may
> > >>>> > > have
> > >>>> > > > to
> > >>>> > > > > > > deal
> > >>>> > > > > > > > > with
> > >>>> > > > > > > > > > > > transactional semantic mismatches across
> > >>>> different
> > >>>> > > > > > persistence
> > >>>> > > > > > > > > > > backends.
> > >>>> > > > > > > > > > > > For example, we would likely avoid using
> > JDBC's
> > >>>> > > > > > > > > > `runWithinTransaction`
> > >>>> > > > > > > > > > > > for
> > >>>> > > > > > > > > > > > single row updates, which adds additional
> > >>>> overhead and
> > >>>> > > > > > > complexity
> > >>>> > > > > > > > > > > > without
> > >>>> > > > > > > > > > > > benefits.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > References:
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > 1.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > >
> > >>>> > > > > >
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>>
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > >>>> > > > > > > > > > > > 2.
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > >
> > >>>> > > > > >
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>>
> >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > Yufei
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri
> > >>>> Bourlatchkov <
> > >>>> > > > > > > > > dimas@apache.org>
> > >>>> > > > > > > > > > > > wrote:
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > Hi all,
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a
> > couple
> > >>>> of
> > >>>> > > > > > interesting
> > >>>> > > > > > > PRs:
> > >>>> > > > > > > > > > > > > [4939], [5035].
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > It looks like people are starting to
> encounter
> > >>>> > > > consistency
> > >>>> > > > > > > issues
> > >>>> > > > > > > > > in
> > >>>> > > > > > > > > > > > > JDBC persistence.
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > The PRs provide valuable insight into the
> > >>>> underlying
> > >>>> > > > > issues.
> > >>>> > > > > > > They
> > >>>> > > > > > > > > > offer
> > >>>> > > > > > > > > > > > > incremental fixes that can work. However, I
> > >>>> believe it
> > >>>> > > is
> > >>>> > > > > > time
> > >>>> > > > > > > for
> > >>>> > > > > > > > > > the
> > >>>> > > > > > > > > > > > > Polaris community to review and improve this
> > >>>> area of
> > >>>> > > the
> > >>>> > > > > > > codebase
> > >>>> > > > > > > > > > > > > holistically.
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > By this, I mean finding a solution that can
> be
> > >>>> applied
> > >>>> > > to
> > >>>> > > > > all
> > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC,
> NoSQL)
> > >>>> and
> > >>>> > > > addresses
> > >>>> > > > > > > these
> > >>>> > > > > > > > > > > > > aspects:
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent
> changes
> > >>>> where
> > >>>> > > the
> > >>>> > > > > > > service
> > >>>> > > > > > > > > > reads
> > >>>> > > > > > > > > > > > > and validates current catalog state, then
> > >>>> commits a
> > >>>> > > > > change
> > >>>> > > > > > > (e.g.
> > >>>> > > > > > > > > > > > > name clashes during renames).
> > >>>> > > > > > > > > > > > > * Supporting consistent but independent
> > changes
> > >>>> to RBAC
> > >>>> > > > > > grants
> > >>>> > > > > > > and
> > >>>> > > > > > > > > > > > > MetaStore entities. This independence is
> > >>>> needed to
> > >>>> > > > > support
> > >>>> > > > > > > > > > > > > external authorizers like OPA and Ranger.
> > >>>> > > > > > > > > > > > > * Supporting atomic changes across multiple
> > >>>> similar
> > >>>> > > > > entities.
> > >>>> > > > > > > > > > > > > * Supporting authorization-based filtering
> of
> > >>>> list
> > >>>> > > > > operations
> > >>>> > > > > > > (cf.
> > >>>> > > > > > > > > > > > > [4831]).
> > >>>> > > > > > > > > > > > > * Supporting credential-vending decisions
> that
> > >>>> are
> > >>>> > > rooted
> > >>>> > > > > in
> > >>>> > > > > > > the
> > >>>> > > > > > > > > > > > > exact state of the catalog.
> > >>>> > > > > > > > > > > > > * Supporting server-side retries for
> transient
> > >>>> > > > persistence
> > >>>> > > > > > > failures
> > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures).
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > Please share your comments and ideas.
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > [4831]
> > >>>> https://github.com/apache/polaris/pull/4831
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > [4939]
> > >>>> https://github.com/apache/polaris/pull/4939
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > [5035]
> > >>>> https://github.com/apache/polaris/pull/5035
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > > > Thanks,
> > >>>> > > > > > > > > > > > > Dmitri
> > >>>> > > > > > > > > > > > >
> > >>>> > > > > > > > > > > >
> > >>>> > > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > >
> > >>>> > > > > >
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>>
> > >>>
> >
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by EJ Wang <ej...@gmail.com>.
Thanks for the recap, Dmitri. I'd like to follow up on the three points I
raised during the sync. We didn't have much time to discuss them, so it
would help to confirm these before choosing an SPI design.
1. Which backend assumptions should we design for?
Within one Polaris server instance, do we assume entities, grants and
secrets all use the same database backend, or must we support mixing
backends across them?
Must every supported persistence implementation provide transactional
guarantees? Does the database itself need to provide transactions, or can
Polaris build that capability on top of simpler database operations?
I'm happy to work with a homogeneous assumption if that's the intended
scope. Existing NoSQL illustrates the second question: it can publish
changes to multiple objects under one reference using a single conditional
pointer update.
2. How do we protect the data used by validation?
Your storage-configuration and location-overlap examples make the problem
concrete. Consider these races:
A table creation passes validation against the catalog's allowed locations,
then another request changes that configuration before the table is
committed.
Two table-creation requests each check for overlapping locations, both see
no overlap, and then create tables at conflicting locations.
Checking only the record being written cannot cover these dependencies. The
first also depends on the catalog configuration. The second depends on the
absence of another overlapping table.
Could the persistence layer protect these reads together with the writes,
using backend-internal versions/tokens or database isolation? Shared code
would still need to identify the required reads and checks. Could we keep
the mechanism for protecting them inside each backend?
3. Business rules are duplicated across backend implementations
Today, the TreeMap
<https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481>
and JDBC
<https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046>
implementations each load a secret, check its principal, apply the
rotation/reset steps and write it back.
Changing these business rules therefore requires keeping multiple
implementations in sync. A fix applied to one backend can be missed in
another, causing the same Polaris operation to behave differently depending
on the database.
My proposal is to implement each business workflow once, against a common
set of storage operations with defined consistency and failure semantics.
The shared workflow should not need to know whether those operations are
implemented through JDBC transactions, NoSQL reference CAS or another
mechanism. Each backend would implement the same storage contract, keeping
its execution details internal. A new database backend could then reuse the
existing business workflows.
Do these seem like reasonable starting points? We can then evaluate the SPI
against the original failure cases on both JDBC and existing NoSQL.
-ej
On Fri, Sep 18, 2026 at 12:10 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Side note: I just came across [5541], which shows a case where a failed
> request leaves persisted side effects.
>
> This can be seen as a coding mistake, of course. However, this is about
> executing one request as a single, atomic unit. This is something that
> probably needs foundational support and coding patterns in Polaris code.
>
> Cheers,
> Dmitri.
> [5541] https://github.com/apache/polaris/pull/5541
>
> On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > Here's a recap of the Community Sync discussion as I remember it. I'm
> sure
> > my recollection is not complete, so please add your thoughts to this
> thread.
> >
> > * Polaris traditionally relies on entity version number to ensure
> > consistency of changes.
> >
> > This works well for internal RBAC grants, which also have version numbers
> > cross-referenced at the Persistence layer.
> >
> > I do not think this works in more general cases like validating locations
> > wrt catalog-level storage configuration.
> >
> > Another difficult case involves location overlaps between concurrent
> table
> > creations.
> >
> > * We talked about how Persistence calls related to RDBMS transactions (in
> > the JDBC persistence case).
> >
> > There were some concerns raised about whether we need to bind Persistence
> > API to transactions explicitly.
> >
> > I think it is a valid concern. At the same time, the Persistence API
> > should be clear about consistency guarantees across all backend
> > implementations. I think this needs more attention now that the matter of
> > milti-entioty changes became prominent in [5035].
> >
> > I personally believe that RDBMS-based Persistence in Polaris has to use
> > Serializable transaction isolation. Consequently, the core code needs to
> > provide mechanisms to allow the Persistence implementation to properly
> > connect reads and writes from API requests to JDBC transactions...
> whether
> > we use "transation" as an explicit terms in the SPI or not.
> >
> > Note that currently, the same API request performs reads and writes in
> > _separate_ RDBMS transactions (even separate connections).
> >
> > * We did not discuss Persistence SPI implications in the call, but
> > currently each Persistence call is assumed to be one distinct and atomic
> > change (please correct me if I'm wrong).
> >
> > Consequently, supporting new use cases involves introducing new SPI
> > methods with many parameters. This can be seen in PR [5035]. This tends
> to
> > overcomplicate the SPI.
> >
> > Ideally, I think the SPI should be simplified to deal with multi-entity
> > and single-entity changes in a coherent manner to avoid any ambiguity
> about
> > which method the caller should use.
> >
> > [5035] https://github.com/apache/polaris/pull/5035
> >
> > Cheers,
> > Dmitri.
> >
> > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> >> Heads up: This discussion is on the Community Sync call agenda for Sept
> >> 17.
> >>
> >> Interested parties, please plan to attend, if possible :)
> >>
> >> Cheers,
> >> Dmitri.
> >>
> >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <di...@apache.org>
> >> wrote:
> >>
> >>> Hi JB,
> >>>
> >>> I've added an agenda item to the next community sync call for this
> (Sept
> >>> 17, 2026).
> >>>
> >>> Re: key points: did you mean a real GH discussion or dev email (this
> >>> thread)? Just double checking :) I'm fine with either approach.
> >>>
> >>> Cheers,
> >>> Dmitri.
> >>>
> >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> >>> wrote:
> >>>
> >>>> Hi Dmitri,
> >>>>
> >>>> I agree on the need to align on these points, though I'm not entirely
> >>>> sure a dedicated meeting is necessary. Let's start by using some time
> >>>> during the next community meeting to discuss it.
> >>>>
> >>>> In the meantime, what if we outline the key points in a GitHub
> >>>> Discussion first? We can then use that as a reference during the
> >>>> meeting.
> >>>>
> >>>> Regards,
> >>>> JB
> >>>>
> >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <dimas@apache.org
> >
> >>>> wrote:
> >>>> >
> >>>> > Hi All,
> >>>> >
> >>>> > My impression from this thread is that it might be time for a call
> to
> >>>> > discuss all the related issues and try to arrive at a shared
> >>>> implementation
> >>>> > plan.
> >>>> >
> >>>> > I know meetings are not ideal, but in this case it might be
> >>>> beneficial as a
> >>>> > means for achieving a common understanding of the set of problems
> and
> >>>> > priorities related to this thread. A dedicated metrics meeting
> worked
> >>>> well
> >>>> > from my POV.
> >>>> >
> >>>> > Allocating a time slot in the next community meeting might be an
> >>>> option,
> >>>> > although I think we might need the full hour in this case.
> >>>> >
> >>>> > WDYT?
> >>>> >
> >>>> > Thanks,
> >>>> > Dmitri.
> >>>> >
> >>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <
> >>>> prithvisivasankar@gmail.com>
> >>>> > wrote:
> >>>> >
> >>>> > > Hi all,
> >>>> > >
> >>>> > > I rewrote https://github.com/apache/polaris/pull/5035, following
> >>>> the
> >>>> > > discussion to focus only on the SPI foundation.
> >>>> > >
> >>>> > > What is in the PR now:
> >>>> > >
> >>>> > > - A written manager-level consistency contract in
> >>>> > > site/content/in-dev/unreleased/persistence-consistency.md.
> >>>> > > - CAS-aware EntityMutation / GrantMutation records.
> >>>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet, with
> >>>> backends
> >>>> > > opting in via supportsAtomicMixedCommit().
> >>>> > > - commitChangeSet implementations for the in-memory TreeMap
> >>>> backend and
> >>>> > > JDBC.
> >>>> > > - AtomicOperationMetaStoreManager and
> >>>> TransactionalMetaStoreManagerImpl
> >>>> > > grant/revoke paths now build one change set per operation,
> >>>> falling back
> >>>> > > to
> >>>> > > individual operations when the backend does not support mixed
> >>>> commits.
> >>>> > > - Tests for the change-set API and TreeMap atomic commits.
> >>>> > >
> >>>> > > I intentionally left out:
> >>>> > >
> >>>> > > - Entity deletes in MetaStoreChangeSet.
> >>>> > > - Refactoring createCatalog, dropEntity, and renameEntity to
> use
> >>>> change
> >>>> > > sets.
> >>>> > > - NoSQL support for commitChangeSet.
> >>>> > >
> >>>> > > I am treating this as Phase 1 so the contract and the SPI
> primitive
> >>>> can be
> >>>> > > reviewed before the larger operation migrations land.
> >>>> > >
> >>>> > > Could you please take a look? In particular, I would like to know
> >>>> whether
> >>>> > > the contract captures the consensus so far, or if it needs to
> >>>> address retry
> >>>> > > / external-work coordination before we merge this foundation.
> >>>> > >
> >>>> > > Thanks,
> >>>> > > Prithvi S
> >>>> > >
> >>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <
> >>>> dimas@apache.org>
> >>>> > > wrote:
> >>>> > >
> >>>> > > > Hi Robert,
> >>>> > > >
> >>>> > > > I agree that the client-visible operation (e.g. REST API
> request)
> >>>> > > consists
> >>>> > > > of many distinct phases. The database / persistence changes are
> >>>> just one
> >>>> > > of
> >>>> > > > these phases. STS (as an example) is another. My point about
> >>>> transactions
> >>>> > > > in JDBC Persistence applies to the former (database changes)
> >>>> phase - one
> >>>> > > > retry attempt there should ideally involve exactly one
> >>>> transaction.
> >>>> > > >
> >>>> > > > We certainly need logic inside Polaris Servers to to coordinate
> >>>> requests
> >>>> > > to
> >>>> > > > various external systems depending on outcomes from previous
> >>>> request
> >>>> > > > processing phases.
> >>>> > > >
> >>>> > > > I also agree that validation based on database state should be
> >>>> repeated
> >>>> > > > from scratch if we retry the database changes.
> >>>> > > >
> >>>> > > > However, internal RBAC authorization naturally has to happen
> >>>> inside the
> >>>> > > > database update phase (read-check-write). IIRC, that is part of
> >>>> the
> >>>> > > > implicit consistency guarantees, which were discussed during
> early
> >>>> > > project
> >>>> > > > months (but never got written down in full clarity,
> >>>> unfortunately). If
> >>>> > > the
> >>>> > > > Authorization check is inside that phase for internal RBAC, I
> >>>> suppose it
> >>>> > > > will be there for other Authorizers too and that can involve
> >>>> external
> >>>> > > > systems (e.g. OPA / Ranger). We can certainly expect
> >>>> authorization to be
> >>>> > > > efficient, but I'm not sure how quick it can be in practice.
> >>>> There's
> >>>> > > > certainly potential for delays in some (perhaps infrequent)
> cases.
> >>>> > > >
> >>>> > > > Cheers,
> >>>> > > > Dmitri.
> >>>> > > >
> >>>> > > >
> >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <sn...@snazy.de>
> >>>> wrote:
> >>>> > > >
> >>>> > > > > Hi Dmitri,
> >>>> > > > >
> >>>> > > > > I agree that one logical operation needs a consistent view of
> >>>> the whole
> >>>> > > > > backend state.
> >>>> > > > > What worries me is treating one long JDBC transaction as the
> >>>> boundary
> >>>> > > of
> >>>> > > > > that operation and then retrying the complete request if the
> >>>> commit
> >>>> > > > fails.
> >>>> > > > >
> >>>> > > > > The request can also write metadata to object storage or call
> >>>> STS.
> >>>> > > > > Each system can leave us with an outcome whose certainty we
> >>>> cannot
> >>>> > > know.
> >>>> > > > > The database may have committed before the connection was
> lost,
> >>>> an
> >>>> > > > > object-store write may have succeeded before a timeout, or STS
> >>>> may have
> >>>> > > > > issued credentials before its response was lost.
> >>>> > > > > A database rollback cannot undo any of those effects, and
> >>>> retrying the
> >>>> > > > > whole request may repeat them.
> >>>> > > > >
> >>>> > > > > So I think the contract has to separate the overall operation
> >>>> from each
> >>>> > > > > attempt to commit it.
> >>>> > > > > Each database or backend commit attempt should stay short.
> >>>> > > > > When we retry, validation and authorization need to use the
> >>>> state for
> >>>> > > > that
> >>>> > > > > new attempt, and we need clear rules for which external work
> >>>> can be
> >>>> > > > reused,
> >>>> > > > > repeated, cleaned up, or reconciled after an uncertain
> outcome.
> >>>> > > > >
> >>>> > > > > Some contextual state could be useful for carrying that state.
> >>>> > > > > But the JDBC connection and transaction should be an
> >>>> implementation
> >>>> > > > detail
> >>>> > > > > inside it, not the boundary of the complete operation.
> >>>> > > > >
> >>>> > > > > Cheers,
> >>>> > > > > Robert
> >>>> > > > >
> >>>> > > > >
> >>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <
> >>>> dimas@apache.org
> >>>> > > >
> >>>> > > > > wrote:
> >>>> > > > >
> >>>> > > > > > Hi All,
> >>>> > > > > >
> >>>> > > > > > The idea of a backend-agnostic change-set primitive sounds
> >>>> good.
> >>>> > > > > >
> >>>> > > > > > However, I am not sure it is sufficient for all the use
> cases
> >>>> we
> >>>> > > > touched
> >>>> > > > > > here. More specifically, the state read by validation code
> is
> >>>> not
> >>>> > > > > > necessarily reflected in the change set:
> >>>> > > > > >
> >>>> > > > > > * Unchanged entities might be considered by validation, but
> >>>> changed
> >>>> > > in
> >>>> > > > a
> >>>> > > > > > parallel request.
> >>>> > > > > > * Even for changed entities the state read by validation may
> >>>> differ
> >>>> > > > from
> >>>> > > > > > the state being altered. Some validation code talks to the
> >>>> MetaStore
> >>>> > > > > > directly, outside of the data produced by the Resolver.
> >>>> > > > > >
> >>>> > > > > > I do not think Polaris offers any explicit mechanisms (ATM)
> >>>> to ensure
> >>>> > > > > > consistency between these reads and subsequent writes.
> >>>> > > > > >
> >>>> > > > > > As far as JDBC goes, running an overarching Transaction
> >>>> across all
> >>>> > > > reads
> >>>> > > > > > and writes in the same request, at the SERIALIZABLE
> isolation
> >>>> level
> >>>> > > in
> >>>> > > > > the
> >>>> > > > > > backing RDBMS could solve the problem, I think.
> >>>> > > > > >
> >>>> > > > > > Indeed, such a transaction might be long to accommodate
> calls
> >>>> made by
> >>>> > > > > > Polaris to external storage, etc. However, is that a
> problem?
> >>>> I think
> >>>> > > > the
> >>>> > > > > > alternative is for the JDBC Persistence impl. to "manually"
> >>>> track all
> >>>> > > > > reads
> >>>> > > > > > under the same request and redo them in the small
> transaction
> >>>> that
> >>>> > > > > persists
> >>>> > > > > > the writes. This will also add RDBMS overhead and require
> >>>> complex
> >>>> > > code
> >>>> > > > in
> >>>> > > > > > Polaris to handle the data properly.
> >>>> > > > > >
> >>>> > > > > > Tracking the request-wide transaction can be done only for
> >>>> JDBC
> >>>> > > without
> >>>> > > > > > leaking "transaction" concepts to the NoSQL Persistence, I
> >>>> think.
> >>>> > > NoSQL
> >>>> > > > > > will use other mechanisms to ensure read/write consistency.
> >>>> > > > > >
> >>>> > > > > > Connecting to Robert's email (a parallel branch in this
> >>>> discussion),
> >>>> > > > I'd
> >>>> > > > > > like to propose this approach:
> >>>> > > > > >
> >>>> > > > > > * Each request establishes a "Data Context"
> >>>> > > > > > - In JDBC the Data Context corresponds to a JDBC
> Connection
> >>>> + Tx
> >>>> > > > > > - In NoSQL the Data Context tracks one or more reference
> >>>> hashes
> >>>> > > > > > * All Persistence access in the same request goes through
> the
> >>>> same
> >>>> > > Data
> >>>> > > > > > Context
> >>>> > > > > > * All Persistence changes are committed once at the end of
> the
> >>>> > > request
> >>>> > > > > > - Not all changes have to be globally atomic. For example,
> >>>> changes
> >>>> > > in
> >>>> > > > > > Catalogs A and B do not have to be atomic with respect to
> >>>> each other.
> >>>> > > > We
> >>>> > > > > > can go deeper into this later. This is relevant to NoSQL.
> >>>> > > > > > - Transactional backends like JDBC can, of course, choose
> >>>> to make
> >>>> > > all
> >>>> > > > > > changes globally atomic.
> >>>> > > > > > * A commit can fail in two main ways:
> >>>> > > > > > - A retriable failure like an optimistic lock error or Tx
> >>>> > > > > serializability
> >>>> > > > > > error
> >>>> > > > > > - A non-triable logical error (e.g. entity not found)
> >>>> > > > > > * On a retriable error the whole request is re-attempted (as
> >>>> if
> >>>> > > > > resubmitted
> >>>> > > > > > by a client) a few times (configurable timeout).
> >>>> > > > > >
> >>>> > > > > > WDYT?
> >>>> > > > > >
> >>>> > > > > > Cheers,
> >>>> > > > > > Dmitri.
> >>>> > > > > >
> >>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré <
> >>>> > > jb@nanthrax.net
> >>>> > > > >
> >>>> > > > > > wrote:
> >>>> > > > > >
> >>>> > > > > > > Hi all
> >>>> > > > > > >
> >>>> > > > > > > I think Robert has a good point.
> >>>> > > > > > >
> >>>> > > > > > > If the atomicity guarantee lives only in BasePersistence,
> >>>> then the
> >>>> > > > > > > manager contract can't tell a caller whether the state it
> >>>> read for
> >>>> > > > > > > validation/authorization/credential-vending is the same
> >>>> state that
> >>>> > > > > > > eventually commits. That's the actual bug class behind the
> >>>> JDBC
> >>>> > > > > > > symptoms (not any single operation being non-atomic, but
> the
> >>>> > > contract
> >>>> > > > > > > being silent about it. Every operation-specific fix (like
> >>>> #4939,
> >>>> > > > #5035
> >>>> > > > > > > or #5095) re-answers this question locally and it keeps
> >>>> recurring.
> >>>> > > So
> >>>> > > > > > > the deliverable should start with a written consistency
> >>>> contract at
> >>>> > > > > > > the manager level.
> >>>> > > > > > >
> >>>> > > > > > > I'm not sure migrating everyone to
> >>>> > > TransactionalMetaStoreManagerImpl
> >>>> > > > > > > is a good idea. It would tie the logical change set to a
> DB
> >>>> > > > > > > transaction spanning the REST request. It means:
> >>>> > > > > > > - it holds a durable transaction open across slow external
> >>>> work
> >>>> > > > > > > (credential vending, OPA, Ranger, ...)
> >>>> > > > > > > - it doesn't map to NoSQL
> >>>> > > > > > > - It wraps single-row updates in runWiithinTransaction,
> >>>> which is a
> >>>> > > > > > overhead
> >>>> > > > > > >
> >>>> > > > > > > So, I think the transactional manager isn't a portable
> >>>> target. It's
> >>>> > > > > > > "only" one backend's strategy.
> >>>> > > > > > >
> >>>> > > > > > > I think Privthi's approach is right. A backend-agnostic
> >>>> change-set
> >>>> > > > > > > primitive with a documented fallback is the correct shape.
> >>>> There is
> >>>> > > > > > > one caveat: without carrying the original entitiy (not
> just
> >>>> the new
> >>>> > > > > > > one) the change set can't express optimistic concurrency.
> >>>> > > > > > >
> >>>> > > > > > > I propose the following multi-steps approach:
> >>>> > > > > > > 1. We write the manager-level consistency contract as
> >>>> Robert asked.
> >>>> > > > It
> >>>> > > > > > > should include the explicit statement that a logical
> change
> >>>> set is
> >>>> > > > not
> >>>> > > > > > > a request-scope DB transaction.
> >>>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency
> pattern)
> >>>> > > baseline
> >>>> > > > > > > first-class in EntityMutation before merging the SPI
> >>>> > > > > > > 3. Then we refactor createCatalog/dropEntity/renameEntity.
> >>>> > > > > > >
> >>>> > > > > > > Thoughts?
> >>>> > > > > > >
> >>>> > > > > > > Regards
> >>>> > > > > > > JB
> >>>> > > > > > >
> >>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <
> >>>> snazy@snazy.de>
> >>>> > > wrote:
> >>>> > > > > > > >
> >>>> > > > > > > > Hi all,
> >>>> > > > > > > >
> >>>> > > > > > > > Yufei's clarification about where the atomicity
> guarantee
> >>>> is
> >>>> > > > defined
> >>>> > > > > > > seems
> >>>> > > > > > > > important.
> >>>> > > > > > > > If it is a property of the lower-level BasePersistence
> >>>> contract
> >>>> > > > > rather
> >>>> > > > > > > than
> >>>> > > > > > > > the general PolarisMetaStoreManager contract, the
> general
> >>>> > > contract
> >>>> > > > > does
> >>>> > > > > > > not
> >>>> > > > > > > > tell callers whether the state used for validation,
> >>>> > > authorization,
> >>>> > > > or
> >>>> > > > > > > > credential vending is consistent with the change that
> >>>> eventually
> >>>> > > > > > commits.
> >>>> > > > > > > >
> >>>> > > > > > > > The current PRs suggest that operation-specific
> >>>> multi-object
> >>>> > > > methods
> >>>> > > > > > can
> >>>> > > > > > > > fix individual cases, while leaving the same contract
> >>>> question to
> >>>> > > > > recur
> >>>> > > > > > > for
> >>>> > > > > > > > each new case.
> >>>> > > > > > > >
> >>>> > > > > > > > I would also avoid defining a logical change set as a
> >>>> database
> >>>> > > > > > > transaction
> >>>> > > > > > > > around an entire REST request.
> >>>> > > > > > > > That would tie the contract to the backend and database,
> >>>> and
> >>>> > > could
> >>>> > > > > keep
> >>>> > > > > > > the
> >>>> > > > > > > > durable attempt open across slow external work.
> >>>> > > > > > > >
> >>>> > > > > > > > So is the choice really between the two current manager
> >>>> > > > > > implementations,
> >>>> > > > > > > or
> >>>> > > > > > > > do we first need to revisit the boundary and guarantees
> >>>> exposed
> >>>> > > to
> >>>> > > > > > their
> >>>> > > > > > > > callers?
> >>>> > > > > > > >
> >>>> > > > > > > > Cheers,
> >>>> > > > > > > > Robert
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov <
> >>>> > > > > dimas@apache.org
> >>>> > > > > > >
> >>>> > > > > > > > wrote:
> >>>> > > > > > > >
> >>>> > > > > > > > > Hi Yufei,
> >>>> > > > > > > > >
> >>>> > > > > > > > > Thanks for the link. I stand corrected. The "one
> atomic
> >>>> change
> >>>> > > > per
> >>>> > > > > > > method"
> >>>> > > > > > > > > contract as defined in javadoc does apply only to
> >>>> > > BasePersistence
> >>>> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates
> to
> >>>> > > > > > > BasePersistence).
> >>>> > > > > > > > >
> >>>> > > > > > > > > Note that all non-test Persistence implementations in
> >>>> the
> >>>> > > Polaris
> >>>> > > > > > > codebase
> >>>> > > > > > > > > extend those classes (which is probably why I was
> >>>> confused
> >>>> > > about
> >>>> > > > > > > atomicity
> >>>> > > > > > > > > expectations).
> >>>> > > > > > > > >
> >>>> > > > > > > > > However, this creates a gap in the Persistence SPI
> >>>> > > specification.
> >>>> > > > > If
> >>>> > > > > > > > > other PolarisMetaStoreManager implementations do not
> >>>> have to
> >>>> > > > comply
> >>>> > > > > > > with
> >>>> > > > > > > > > this principle, it will create a conceptual difficulty
> >>>> at call
> >>>> > > > > sites.
> >>>> > > > > > > How
> >>>> > > > > > > > > can PolarisMetaStoreManager callers reason about
> >>>> consistency
> >>>> > > and
> >>>> > > > > > > durability
> >>>> > > > > > > > > behaviours in general?
> >>>> > > > > > > > >
> >>>> > > > > > > > > I believe we need to address that as part of this
> >>>> discussion.
> >>>> > > > > > > > >
> >>>> > > > > > > > > Cheers,
> >>>> > > > > > > > > Dmitri.
> >>>> > > > > > > > >
> >>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <
> >>>> flyrain000@gmail.com
> >>>> > > >
> >>>> > > > > > wrote:
> >>>> > > > > > > > >
> >>>> > > > > > > > > > > The MetaStore SPI is currently defined with the
> >>>> idea that
> >>>> > > one
> >>>> > > > > > > method
> >>>> > > > > > > > > call
> >>>> > > > > > > > > > means one atomic change.
> >>>> > > > > > > > > >
> >>>> > > > > > > > > > If the "MetaStore SPI" refers to the interface
> >>>> > > > > > > PolarisMetaStoreManager, I
> >>>> > > > > > > > > > don't think we've ever state each method to be
> >>>> atomic. We did
> >>>> > > > > > clarify
> >>>> > > > > > > > > > atomicity[1] in the interface BasePersistence
> though.
> >>>> > > > > > > > > >
> >>>> > > > > > > > > >
> >>>> > > > > > > > > > 1.
> >>>> > > > > > > > > >
> >>>> > > > > > > > > >
> >>>> > > > > > > > >
> >>>> > > > > > >
> >>>> > > > > >
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>>
> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> >>>> > > > > > > > > >
> >>>> > > > > > > > > >
> >>>> > > > > > > > > >
> >>>> > > > > > > > > > Yufei
> >>>> > > > > > > > > >
> >>>> > > > > > > > > >
> >>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri
> Bourlatchkov <
> >>>> > > > > > > dimas@apache.org>
> >>>> > > > > > > > > > wrote:
> >>>> > > > > > > > > >
> >>>> > > > > > > > > > > Hi Yufei,
> >>>> > > > > > > > > > >
> >>>> > > > > > > > > > > I agree that JDBC transactions must be handled
> more
> >>>> > > > explicitly.
> >>>> > > > > > > > > However,
> >>>> > > > > > > > > > > I'm not sure that simply moving to
> >>>> > > > > > > TransactionalMetaStoreManagerImpl is
> >>>> > > > > > > > > > > sufficient.
> >>>> > > > > > > > > > >
> >>>> > > > > > > > > > > The MetaStore SPI is currently defined with the
> >>>> idea that
> >>>> > > one
> >>>> > > > > > > method
> >>>> > > > > > > > > call
> >>>> > > > > > > > > > > means one atomic change [1]. The "transactional"
> >>>> MetaStore
> >>>> > > > > impl.
> >>>> > > > > > is
> >>>> > > > > > > > > but a
> >>>> > > > > > > > > > > sub-case of that. It cannot alter the high-level
> >>>> contract.
> >>>> > > > > > > > > > >
> >>>> > > > > > > > > > > We could add SPI methods having multiple object
> >>>> parameters
> >>>> > > to
> >>>> > > > > > > represent
> >>>> > > > > > > > > > > grouped changes, but I am not sure it will be a
> >>>> sound
> >>>> > > design.
> >>>> > > > > > This
> >>>> > > > > > > will
> >>>> > > > > > > > > > > bloat the interface surfaces and require extra
> >>>> impl. effort
> >>>> > > > for
> >>>> > > > > > > each
> >>>> > > > > > > > > > > backend type. More importantly, adding multi-arg
> >>>> change
> >>>> > > > methods
> >>>> > > > > > > still
> >>>> > > > > > > > > > won't
> >>>> > > > > > > > > > > address the problem of reads being consistent with
> >>>> writes,
> >>>> > > > > > because
> >>>> > > > > > > each
> >>>> > > > > > > > > > > method call will still be independent regarding
> the
> >>>> data
> >>>> > > > stored
> >>>> > > > > > in
> >>>> > > > > > > the
> >>>> > > > > > > > > > > database.
> >>>> > > > > > > > > > >
> >>>> > > > > > > > > > > I tend to think we need to introduce a "change
> set"
> >>>> or
> >>>> > > > "atomic
> >>>> > > > > > > batch"
> >>>> > > > > > > > > > > concept to core Persistence and associate each
> REST
> >>>> API
> >>>> > > > request
> >>>> > > > > > > with
> >>>> > > > > > > > > one
> >>>> > > > > > > > > > > such change set, which will be committed (or
> rolled
> >>>> back)
> >>>> > > at
> >>>> > > > > the
> >>>> > > > > > > end of
> >>>> > > > > > > > > > the
> >>>> > > > > > > > > > > request. I believe Ayush mentioned a similar
> >>>> concept in PR
> >>>> > > > 4939
> >>>> > > > > > > [2]. In
> >>>> > > > > > > > > > > JDBC each change set will naturally be associated
> >>>> with an
> >>>> > > > RDBMS
> >>>> > > > > > > > > > > transaction. In NoSQL persistence, each atomic
> >>>> change set
> >>>> > > > will
> >>>> > > > > be
> >>>> > > > > > > > > > > associated with one CAS operation on the
> underlying
> >>>> > > database.
> >>>> > > > > > > > > > >
> >>>> > > > > > > > > > > [1]
> >>>> > > > > > >
> >>>> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
> >>>> > > > > > > > > > >
> >>>> > > > > > > > > > > [2]
> >>>> > > > > > >
> >>>> https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> >>>> > > > > > > > > > >
> >>>> > > > > > > > > > > Cheers,
> >>>> > > > > > > > > > > Dmitri.
> >>>> > > > > > > > > > >
> >>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
> >>>> > > > flyrain000@gmail.com
> >>>> > > > > >
> >>>> > > > > > > wrote:
> >>>> > > > > > > > > > >
> >>>> > > > > > > > > > > > Thanks for raising this, Dmitri.
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > > These are valid concerns, and they were already
> >>>> > > recognized
> >>>> > > > > when
> >>>> > > > > > > we
> >>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At that
> >>>> time, we
> >>>> > > > > chose
> >>>> > > > > > > to use
> >>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC due
> >>>> to the
> >>>> > > > > > > simplicity. I
> >>>> > > > > > > > > > > think
> >>>> > > > > > > > > > > > most of the issues mentioned here can already be
> >>>> > > addressed
> >>>> > > > by
> >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl.
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > > For example, rename is already wrapped in a
> >>>> transaction
> >>>> > > in
> >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1].
> Similarly,
> >>>> catalog
> >>>> > > > > > > creation,
> >>>> > > > > > > > > > which
> >>>> > > > > > > > > > > > involves reading and creating multiple objects,
> >>>> is also
> >>>> > > > > > executed
> >>>> > > > > > > > > > within a
> >>>> > > > > > > > > > > > transaction [2].
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > > I see two possible directions:
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > > 1.
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager
> >>>> together with
> >>>> > > the
> >>>> > > > > > > > > persistence
> >>>> > > > > > > > > > > > backends (such as JDBC) to provide the
> required
> >>>> > > > > consistency
> >>>> > > > > > > > > > guarantees
> >>>> > > > > > > > > > > > for
> >>>> > > > > > > > > > > > specific operations, similar to what
> >>>> > > > > > > > > > TransactionalMetaStoreManagerImpl
> >>>> > > > > > > > > > > > does.
> >>>> > > > > > > > > > > > 2.
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > > Migrate the persistence backends (such as
> >>>> JDBC) to use
> >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl directly.
> We
> >>>> may
> >>>> > > have
> >>>> > > > to
> >>>> > > > > > > deal
> >>>> > > > > > > > > with
> >>>> > > > > > > > > > > > transactional semantic mismatches across
> >>>> different
> >>>> > > > > > persistence
> >>>> > > > > > > > > > > backends.
> >>>> > > > > > > > > > > > For example, we would likely avoid using
> JDBC's
> >>>> > > > > > > > > > `runWithinTransaction`
> >>>> > > > > > > > > > > > for
> >>>> > > > > > > > > > > > single row updates, which adds additional
> >>>> overhead and
> >>>> > > > > > > complexity
> >>>> > > > > > > > > > > > without
> >>>> > > > > > > > > > > > benefits.
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > > References:
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > > 1.
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > >
> >>>> > > > > > > > > >
> >>>> > > > > > > > >
> >>>> > > > > > >
> >>>> > > > > >
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>>
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> >>>> > > > > > > > > > > > 2.
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > >
> >>>> > > > > > > > > >
> >>>> > > > > > > > >
> >>>> > > > > > >
> >>>> > > > > >
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>>
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > > Yufei
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri
> >>>> Bourlatchkov <
> >>>> > > > > > > > > dimas@apache.org>
> >>>> > > > > > > > > > > > wrote:
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > > > > Hi all,
> >>>> > > > > > > > > > > > >
> >>>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a
> couple
> >>>> of
> >>>> > > > > > interesting
> >>>> > > > > > > PRs:
> >>>> > > > > > > > > > > > > [4939], [5035].
> >>>> > > > > > > > > > > > >
> >>>> > > > > > > > > > > > > It looks like people are starting to encounter
> >>>> > > > consistency
> >>>> > > > > > > issues
> >>>> > > > > > > > > in
> >>>> > > > > > > > > > > > > JDBC persistence.
> >>>> > > > > > > > > > > > >
> >>>> > > > > > > > > > > > > The PRs provide valuable insight into the
> >>>> underlying
> >>>> > > > > issues.
> >>>> > > > > > > They
> >>>> > > > > > > > > > offer
> >>>> > > > > > > > > > > > > incremental fixes that can work. However, I
> >>>> believe it
> >>>> > > is
> >>>> > > > > > time
> >>>> > > > > > > for
> >>>> > > > > > > > > > the
> >>>> > > > > > > > > > > > > Polaris community to review and improve this
> >>>> area of
> >>>> > > the
> >>>> > > > > > > codebase
> >>>> > > > > > > > > > > > > holistically.
> >>>> > > > > > > > > > > > >
> >>>> > > > > > > > > > > > > By this, I mean finding a solution that can be
> >>>> applied
> >>>> > > to
> >>>> > > > > all
> >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC, NoSQL)
> >>>> and
> >>>> > > > addresses
> >>>> > > > > > > these
> >>>> > > > > > > > > > > > > aspects:
> >>>> > > > > > > > > > > > >
> >>>> > > > > > > > > > > > > * Supporting concurrent and consistent changes
> >>>> where
> >>>> > > the
> >>>> > > > > > > service
> >>>> > > > > > > > > > reads
> >>>> > > > > > > > > > > > > and validates current catalog state, then
> >>>> commits a
> >>>> > > > > change
> >>>> > > > > > > (e.g.
> >>>> > > > > > > > > > > > > name clashes during renames).
> >>>> > > > > > > > > > > > > * Supporting consistent but independent
> changes
> >>>> to RBAC
> >>>> > > > > > grants
> >>>> > > > > > > and
> >>>> > > > > > > > > > > > > MetaStore entities. This independence is
> >>>> needed to
> >>>> > > > > support
> >>>> > > > > > > > > > > > > external authorizers like OPA and Ranger.
> >>>> > > > > > > > > > > > > * Supporting atomic changes across multiple
> >>>> similar
> >>>> > > > > entities.
> >>>> > > > > > > > > > > > > * Supporting authorization-based filtering of
> >>>> list
> >>>> > > > > operations
> >>>> > > > > > > (cf.
> >>>> > > > > > > > > > > > > [4831]).
> >>>> > > > > > > > > > > > > * Supporting credential-vending decisions that
> >>>> are
> >>>> > > rooted
> >>>> > > > > in
> >>>> > > > > > > the
> >>>> > > > > > > > > > > > > exact state of the catalog.
> >>>> > > > > > > > > > > > > * Supporting server-side retries for transient
> >>>> > > > persistence
> >>>> > > > > > > failures
> >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures).
> >>>> > > > > > > > > > > > >
> >>>> > > > > > > > > > > > > Please share your comments and ideas.
> >>>> > > > > > > > > > > > >
> >>>> > > > > > > > > > > > > [4831]
> >>>> https://github.com/apache/polaris/pull/4831
> >>>> > > > > > > > > > > > >
> >>>> > > > > > > > > > > > > [4939]
> >>>> https://github.com/apache/polaris/pull/4939
> >>>> > > > > > > > > > > > >
> >>>> > > > > > > > > > > > > [5035]
> >>>> https://github.com/apache/polaris/pull/5035
> >>>> > > > > > > > > > > > >
> >>>> > > > > > > > > > > > > Thanks,
> >>>> > > > > > > > > > > > > Dmitri
> >>>> > > > > > > > > > > > >
> >>>> > > > > > > > > > > >
> >>>> > > > > > > > > > >
> >>>> > > > > > > > > >
> >>>> > > > > > > > >
> >>>> > > > > > >
> >>>> > > > > >
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>>
> >>>
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Side note: I just came across [5541], which shows a case where a failed
request leaves persisted side effects.
This can be seen as a coding mistake, of course. However, this is about
executing one request as a single, atomic unit. This is something that
probably needs foundational support and coding patterns in Polaris code.
Cheers,
Dmitri.
[5541] https://github.com/apache/polaris/pull/5541
On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi All,
>
> Here's a recap of the Community Sync discussion as I remember it. I'm sure
> my recollection is not complete, so please add your thoughts to this thread.
>
> * Polaris traditionally relies on entity version number to ensure
> consistency of changes.
>
> This works well for internal RBAC grants, which also have version numbers
> cross-referenced at the Persistence layer.
>
> I do not think this works in more general cases like validating locations
> wrt catalog-level storage configuration.
>
> Another difficult case involves location overlaps between concurrent table
> creations.
>
> * We talked about how Persistence calls related to RDBMS transactions (in
> the JDBC persistence case).
>
> There were some concerns raised about whether we need to bind Persistence
> API to transactions explicitly.
>
> I think it is a valid concern. At the same time, the Persistence API
> should be clear about consistency guarantees across all backend
> implementations. I think this needs more attention now that the matter of
> milti-entioty changes became prominent in [5035].
>
> I personally believe that RDBMS-based Persistence in Polaris has to use
> Serializable transaction isolation. Consequently, the core code needs to
> provide mechanisms to allow the Persistence implementation to properly
> connect reads and writes from API requests to JDBC transactions... whether
> we use "transation" as an explicit terms in the SPI or not.
>
> Note that currently, the same API request performs reads and writes in
> _separate_ RDBMS transactions (even separate connections).
>
> * We did not discuss Persistence SPI implications in the call, but
> currently each Persistence call is assumed to be one distinct and atomic
> change (please correct me if I'm wrong).
>
> Consequently, supporting new use cases involves introducing new SPI
> methods with many parameters. This can be seen in PR [5035]. This tends to
> overcomplicate the SPI.
>
> Ideally, I think the SPI should be simplified to deal with multi-entity
> and single-entity changes in a coherent manner to avoid any ambiguity about
> which method the caller should use.
>
> [5035] https://github.com/apache/polaris/pull/5035
>
> Cheers,
> Dmitri.
>
> On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
>> Heads up: This discussion is on the Community Sync call agenda for Sept
>> 17.
>>
>> Interested parties, please plan to attend, if possible :)
>>
>> Cheers,
>> Dmitri.
>>
>> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <di...@apache.org>
>> wrote:
>>
>>> Hi JB,
>>>
>>> I've added an agenda item to the next community sync call for this (Sept
>>> 17, 2026).
>>>
>>> Re: key points: did you mean a real GH discussion or dev email (this
>>> thread)? Just double checking :) I'm fine with either approach.
>>>
>>> Cheers,
>>> Dmitri.
>>>
>>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>>> Hi Dmitri,
>>>>
>>>> I agree on the need to align on these points, though I'm not entirely
>>>> sure a dedicated meeting is necessary. Let's start by using some time
>>>> during the next community meeting to discuss it.
>>>>
>>>> In the meantime, what if we outline the key points in a GitHub
>>>> Discussion first? We can then use that as a reference during the
>>>> meeting.
>>>>
>>>> Regards,
>>>> JB
>>>>
>>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <di...@apache.org>
>>>> wrote:
>>>> >
>>>> > Hi All,
>>>> >
>>>> > My impression from this thread is that it might be time for a call to
>>>> > discuss all the related issues and try to arrive at a shared
>>>> implementation
>>>> > plan.
>>>> >
>>>> > I know meetings are not ideal, but in this case it might be
>>>> beneficial as a
>>>> > means for achieving a common understanding of the set of problems and
>>>> > priorities related to this thread. A dedicated metrics meeting worked
>>>> well
>>>> > from my POV.
>>>> >
>>>> > Allocating a time slot in the next community meeting might be an
>>>> option,
>>>> > although I think we might need the full hour in this case.
>>>> >
>>>> > WDYT?
>>>> >
>>>> > Thanks,
>>>> > Dmitri.
>>>> >
>>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <
>>>> prithvisivasankar@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Hi all,
>>>> > >
>>>> > > I rewrote https://github.com/apache/polaris/pull/5035, following
>>>> the
>>>> > > discussion to focus only on the SPI foundation.
>>>> > >
>>>> > > What is in the PR now:
>>>> > >
>>>> > > - A written manager-level consistency contract in
>>>> > > site/content/in-dev/unreleased/persistence-consistency.md.
>>>> > > - CAS-aware EntityMutation / GrantMutation records.
>>>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet, with
>>>> backends
>>>> > > opting in via supportsAtomicMixedCommit().
>>>> > > - commitChangeSet implementations for the in-memory TreeMap
>>>> backend and
>>>> > > JDBC.
>>>> > > - AtomicOperationMetaStoreManager and
>>>> TransactionalMetaStoreManagerImpl
>>>> > > grant/revoke paths now build one change set per operation,
>>>> falling back
>>>> > > to
>>>> > > individual operations when the backend does not support mixed
>>>> commits.
>>>> > > - Tests for the change-set API and TreeMap atomic commits.
>>>> > >
>>>> > > I intentionally left out:
>>>> > >
>>>> > > - Entity deletes in MetaStoreChangeSet.
>>>> > > - Refactoring createCatalog, dropEntity, and renameEntity to use
>>>> change
>>>> > > sets.
>>>> > > - NoSQL support for commitChangeSet.
>>>> > >
>>>> > > I am treating this as Phase 1 so the contract and the SPI primitive
>>>> can be
>>>> > > reviewed before the larger operation migrations land.
>>>> > >
>>>> > > Could you please take a look? In particular, I would like to know
>>>> whether
>>>> > > the contract captures the consensus so far, or if it needs to
>>>> address retry
>>>> > > / external-work coordination before we merge this foundation.
>>>> > >
>>>> > > Thanks,
>>>> > > Prithvi S
>>>> > >
>>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <
>>>> dimas@apache.org>
>>>> > > wrote:
>>>> > >
>>>> > > > Hi Robert,
>>>> > > >
>>>> > > > I agree that the client-visible operation (e.g. REST API request)
>>>> > > consists
>>>> > > > of many distinct phases. The database / persistence changes are
>>>> just one
>>>> > > of
>>>> > > > these phases. STS (as an example) is another. My point about
>>>> transactions
>>>> > > > in JDBC Persistence applies to the former (database changes)
>>>> phase - one
>>>> > > > retry attempt there should ideally involve exactly one
>>>> transaction.
>>>> > > >
>>>> > > > We certainly need logic inside Polaris Servers to to coordinate
>>>> requests
>>>> > > to
>>>> > > > various external systems depending on outcomes from previous
>>>> request
>>>> > > > processing phases.
>>>> > > >
>>>> > > > I also agree that validation based on database state should be
>>>> repeated
>>>> > > > from scratch if we retry the database changes.
>>>> > > >
>>>> > > > However, internal RBAC authorization naturally has to happen
>>>> inside the
>>>> > > > database update phase (read-check-write). IIRC, that is part of
>>>> the
>>>> > > > implicit consistency guarantees, which were discussed during early
>>>> > > project
>>>> > > > months (but never got written down in full clarity,
>>>> unfortunately). If
>>>> > > the
>>>> > > > Authorization check is inside that phase for internal RBAC, I
>>>> suppose it
>>>> > > > will be there for other Authorizers too and that can involve
>>>> external
>>>> > > > systems (e.g. OPA / Ranger). We can certainly expect
>>>> authorization to be
>>>> > > > efficient, but I'm not sure how quick it can be in practice.
>>>> There's
>>>> > > > certainly potential for delays in some (perhaps infrequent) cases.
>>>> > > >
>>>> > > > Cheers,
>>>> > > > Dmitri.
>>>> > > >
>>>> > > >
>>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <sn...@snazy.de>
>>>> wrote:
>>>> > > >
>>>> > > > > Hi Dmitri,
>>>> > > > >
>>>> > > > > I agree that one logical operation needs a consistent view of
>>>> the whole
>>>> > > > > backend state.
>>>> > > > > What worries me is treating one long JDBC transaction as the
>>>> boundary
>>>> > > of
>>>> > > > > that operation and then retrying the complete request if the
>>>> commit
>>>> > > > fails.
>>>> > > > >
>>>> > > > > The request can also write metadata to object storage or call
>>>> STS.
>>>> > > > > Each system can leave us with an outcome whose certainty we
>>>> cannot
>>>> > > know.
>>>> > > > > The database may have committed before the connection was lost,
>>>> an
>>>> > > > > object-store write may have succeeded before a timeout, or STS
>>>> may have
>>>> > > > > issued credentials before its response was lost.
>>>> > > > > A database rollback cannot undo any of those effects, and
>>>> retrying the
>>>> > > > > whole request may repeat them.
>>>> > > > >
>>>> > > > > So I think the contract has to separate the overall operation
>>>> from each
>>>> > > > > attempt to commit it.
>>>> > > > > Each database or backend commit attempt should stay short.
>>>> > > > > When we retry, validation and authorization need to use the
>>>> state for
>>>> > > > that
>>>> > > > > new attempt, and we need clear rules for which external work
>>>> can be
>>>> > > > reused,
>>>> > > > > repeated, cleaned up, or reconciled after an uncertain outcome.
>>>> > > > >
>>>> > > > > Some contextual state could be useful for carrying that state.
>>>> > > > > But the JDBC connection and transaction should be an
>>>> implementation
>>>> > > > detail
>>>> > > > > inside it, not the boundary of the complete operation.
>>>> > > > >
>>>> > > > > Cheers,
>>>> > > > > Robert
>>>> > > > >
>>>> > > > >
>>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <
>>>> dimas@apache.org
>>>> > > >
>>>> > > > > wrote:
>>>> > > > >
>>>> > > > > > Hi All,
>>>> > > > > >
>>>> > > > > > The idea of a backend-agnostic change-set primitive sounds
>>>> good.
>>>> > > > > >
>>>> > > > > > However, I am not sure it is sufficient for all the use cases
>>>> we
>>>> > > > touched
>>>> > > > > > here. More specifically, the state read by validation code is
>>>> not
>>>> > > > > > necessarily reflected in the change set:
>>>> > > > > >
>>>> > > > > > * Unchanged entities might be considered by validation, but
>>>> changed
>>>> > > in
>>>> > > > a
>>>> > > > > > parallel request.
>>>> > > > > > * Even for changed entities the state read by validation may
>>>> differ
>>>> > > > from
>>>> > > > > > the state being altered. Some validation code talks to the
>>>> MetaStore
>>>> > > > > > directly, outside of the data produced by the Resolver.
>>>> > > > > >
>>>> > > > > > I do not think Polaris offers any explicit mechanisms (ATM)
>>>> to ensure
>>>> > > > > > consistency between these reads and subsequent writes.
>>>> > > > > >
>>>> > > > > > As far as JDBC goes, running an overarching Transaction
>>>> across all
>>>> > > > reads
>>>> > > > > > and writes in the same request, at the SERIALIZABLE isolation
>>>> level
>>>> > > in
>>>> > > > > the
>>>> > > > > > backing RDBMS could solve the problem, I think.
>>>> > > > > >
>>>> > > > > > Indeed, such a transaction might be long to accommodate calls
>>>> made by
>>>> > > > > > Polaris to external storage, etc. However, is that a problem?
>>>> I think
>>>> > > > the
>>>> > > > > > alternative is for the JDBC Persistence impl. to "manually"
>>>> track all
>>>> > > > > reads
>>>> > > > > > under the same request and redo them in the small transaction
>>>> that
>>>> > > > > persists
>>>> > > > > > the writes. This will also add RDBMS overhead and require
>>>> complex
>>>> > > code
>>>> > > > in
>>>> > > > > > Polaris to handle the data properly.
>>>> > > > > >
>>>> > > > > > Tracking the request-wide transaction can be done only for
>>>> JDBC
>>>> > > without
>>>> > > > > > leaking "transaction" concepts to the NoSQL Persistence, I
>>>> think.
>>>> > > NoSQL
>>>> > > > > > will use other mechanisms to ensure read/write consistency.
>>>> > > > > >
>>>> > > > > > Connecting to Robert's email (a parallel branch in this
>>>> discussion),
>>>> > > > I'd
>>>> > > > > > like to propose this approach:
>>>> > > > > >
>>>> > > > > > * Each request establishes a "Data Context"
>>>> > > > > > - In JDBC the Data Context corresponds to a JDBC Connection
>>>> + Tx
>>>> > > > > > - In NoSQL the Data Context tracks one or more reference
>>>> hashes
>>>> > > > > > * All Persistence access in the same request goes through the
>>>> same
>>>> > > Data
>>>> > > > > > Context
>>>> > > > > > * All Persistence changes are committed once at the end of the
>>>> > > request
>>>> > > > > > - Not all changes have to be globally atomic. For example,
>>>> changes
>>>> > > in
>>>> > > > > > Catalogs A and B do not have to be atomic with respect to
>>>> each other.
>>>> > > > We
>>>> > > > > > can go deeper into this later. This is relevant to NoSQL.
>>>> > > > > > - Transactional backends like JDBC can, of course, choose
>>>> to make
>>>> > > all
>>>> > > > > > changes globally atomic.
>>>> > > > > > * A commit can fail in two main ways:
>>>> > > > > > - A retriable failure like an optimistic lock error or Tx
>>>> > > > > serializability
>>>> > > > > > error
>>>> > > > > > - A non-triable logical error (e.g. entity not found)
>>>> > > > > > * On a retriable error the whole request is re-attempted (as
>>>> if
>>>> > > > > resubmitted
>>>> > > > > > by a client) a few times (configurable timeout).
>>>> > > > > >
>>>> > > > > > WDYT?
>>>> > > > > >
>>>> > > > > > Cheers,
>>>> > > > > > Dmitri.
>>>> > > > > >
>>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré <
>>>> > > jb@nanthrax.net
>>>> > > > >
>>>> > > > > > wrote:
>>>> > > > > >
>>>> > > > > > > Hi all
>>>> > > > > > >
>>>> > > > > > > I think Robert has a good point.
>>>> > > > > > >
>>>> > > > > > > If the atomicity guarantee lives only in BasePersistence,
>>>> then the
>>>> > > > > > > manager contract can't tell a caller whether the state it
>>>> read for
>>>> > > > > > > validation/authorization/credential-vending is the same
>>>> state that
>>>> > > > > > > eventually commits. That's the actual bug class behind the
>>>> JDBC
>>>> > > > > > > symptoms (not any single operation being non-atomic, but the
>>>> > > contract
>>>> > > > > > > being silent about it. Every operation-specific fix (like
>>>> #4939,
>>>> > > > #5035
>>>> > > > > > > or #5095) re-answers this question locally and it keeps
>>>> recurring.
>>>> > > So
>>>> > > > > > > the deliverable should start with a written consistency
>>>> contract at
>>>> > > > > > > the manager level.
>>>> > > > > > >
>>>> > > > > > > I'm not sure migrating everyone to
>>>> > > TransactionalMetaStoreManagerImpl
>>>> > > > > > > is a good idea. It would tie the logical change set to a DB
>>>> > > > > > > transaction spanning the REST request. It means:
>>>> > > > > > > - it holds a durable transaction open across slow external
>>>> work
>>>> > > > > > > (credential vending, OPA, Ranger, ...)
>>>> > > > > > > - it doesn't map to NoSQL
>>>> > > > > > > - It wraps single-row updates in runWiithinTransaction,
>>>> which is a
>>>> > > > > > overhead
>>>> > > > > > >
>>>> > > > > > > So, I think the transactional manager isn't a portable
>>>> target. It's
>>>> > > > > > > "only" one backend's strategy.
>>>> > > > > > >
>>>> > > > > > > I think Privthi's approach is right. A backend-agnostic
>>>> change-set
>>>> > > > > > > primitive with a documented fallback is the correct shape.
>>>> There is
>>>> > > > > > > one caveat: without carrying the original entitiy (not just
>>>> the new
>>>> > > > > > > one) the change set can't express optimistic concurrency.
>>>> > > > > > >
>>>> > > > > > > I propose the following multi-steps approach:
>>>> > > > > > > 1. We write the manager-level consistency contract as
>>>> Robert asked.
>>>> > > > It
>>>> > > > > > > should include the explicit statement that a logical change
>>>> set is
>>>> > > > not
>>>> > > > > > > a request-scope DB transaction.
>>>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency pattern)
>>>> > > baseline
>>>> > > > > > > first-class in EntityMutation before merging the SPI
>>>> > > > > > > 3. Then we refactor createCatalog/dropEntity/renameEntity.
>>>> > > > > > >
>>>> > > > > > > Thoughts?
>>>> > > > > > >
>>>> > > > > > > Regards
>>>> > > > > > > JB
>>>> > > > > > >
>>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <
>>>> snazy@snazy.de>
>>>> > > wrote:
>>>> > > > > > > >
>>>> > > > > > > > Hi all,
>>>> > > > > > > >
>>>> > > > > > > > Yufei's clarification about where the atomicity guarantee
>>>> is
>>>> > > > defined
>>>> > > > > > > seems
>>>> > > > > > > > important.
>>>> > > > > > > > If it is a property of the lower-level BasePersistence
>>>> contract
>>>> > > > > rather
>>>> > > > > > > than
>>>> > > > > > > > the general PolarisMetaStoreManager contract, the general
>>>> > > contract
>>>> > > > > does
>>>> > > > > > > not
>>>> > > > > > > > tell callers whether the state used for validation,
>>>> > > authorization,
>>>> > > > or
>>>> > > > > > > > credential vending is consistent with the change that
>>>> eventually
>>>> > > > > > commits.
>>>> > > > > > > >
>>>> > > > > > > > The current PRs suggest that operation-specific
>>>> multi-object
>>>> > > > methods
>>>> > > > > > can
>>>> > > > > > > > fix individual cases, while leaving the same contract
>>>> question to
>>>> > > > > recur
>>>> > > > > > > for
>>>> > > > > > > > each new case.
>>>> > > > > > > >
>>>> > > > > > > > I would also avoid defining a logical change set as a
>>>> database
>>>> > > > > > > transaction
>>>> > > > > > > > around an entire REST request.
>>>> > > > > > > > That would tie the contract to the backend and database,
>>>> and
>>>> > > could
>>>> > > > > keep
>>>> > > > > > > the
>>>> > > > > > > > durable attempt open across slow external work.
>>>> > > > > > > >
>>>> > > > > > > > So is the choice really between the two current manager
>>>> > > > > > implementations,
>>>> > > > > > > or
>>>> > > > > > > > do we first need to revisit the boundary and guarantees
>>>> exposed
>>>> > > to
>>>> > > > > > their
>>>> > > > > > > > callers?
>>>> > > > > > > >
>>>> > > > > > > > Cheers,
>>>> > > > > > > > Robert
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov <
>>>> > > > > dimas@apache.org
>>>> > > > > > >
>>>> > > > > > > > wrote:
>>>> > > > > > > >
>>>> > > > > > > > > Hi Yufei,
>>>> > > > > > > > >
>>>> > > > > > > > > Thanks for the link. I stand corrected. The "one atomic
>>>> change
>>>> > > > per
>>>> > > > > > > method"
>>>> > > > > > > > > contract as defined in javadoc does apply only to
>>>> > > BasePersistence
>>>> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates to
>>>> > > > > > > BasePersistence).
>>>> > > > > > > > >
>>>> > > > > > > > > Note that all non-test Persistence implementations in
>>>> the
>>>> > > Polaris
>>>> > > > > > > codebase
>>>> > > > > > > > > extend those classes (which is probably why I was
>>>> confused
>>>> > > about
>>>> > > > > > > atomicity
>>>> > > > > > > > > expectations).
>>>> > > > > > > > >
>>>> > > > > > > > > However, this creates a gap in the Persistence SPI
>>>> > > specification.
>>>> > > > > If
>>>> > > > > > > > > other PolarisMetaStoreManager implementations do not
>>>> have to
>>>> > > > comply
>>>> > > > > > > with
>>>> > > > > > > > > this principle, it will create a conceptual difficulty
>>>> at call
>>>> > > > > sites.
>>>> > > > > > > How
>>>> > > > > > > > > can PolarisMetaStoreManager callers reason about
>>>> consistency
>>>> > > and
>>>> > > > > > > durability
>>>> > > > > > > > > behaviours in general?
>>>> > > > > > > > >
>>>> > > > > > > > > I believe we need to address that as part of this
>>>> discussion.
>>>> > > > > > > > >
>>>> > > > > > > > > Cheers,
>>>> > > > > > > > > Dmitri.
>>>> > > > > > > > >
>>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <
>>>> flyrain000@gmail.com
>>>> > > >
>>>> > > > > > wrote:
>>>> > > > > > > > >
>>>> > > > > > > > > > > The MetaStore SPI is currently defined with the
>>>> idea that
>>>> > > one
>>>> > > > > > > method
>>>> > > > > > > > > call
>>>> > > > > > > > > > means one atomic change.
>>>> > > > > > > > > >
>>>> > > > > > > > > > If the "MetaStore SPI" refers to the interface
>>>> > > > > > > PolarisMetaStoreManager, I
>>>> > > > > > > > > > don't think we've ever state each method to be
>>>> atomic. We did
>>>> > > > > > clarify
>>>> > > > > > > > > > atomicity[1] in the interface BasePersistence though.
>>>> > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > > > 1.
>>>> > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
>>>> > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > > > Yufei
>>>> > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri Bourlatchkov <
>>>> > > > > > > dimas@apache.org>
>>>> > > > > > > > > > wrote:
>>>> > > > > > > > > >
>>>> > > > > > > > > > > Hi Yufei,
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > I agree that JDBC transactions must be handled more
>>>> > > > explicitly.
>>>> > > > > > > > > However,
>>>> > > > > > > > > > > I'm not sure that simply moving to
>>>> > > > > > > TransactionalMetaStoreManagerImpl is
>>>> > > > > > > > > > > sufficient.
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > The MetaStore SPI is currently defined with the
>>>> idea that
>>>> > > one
>>>> > > > > > > method
>>>> > > > > > > > > call
>>>> > > > > > > > > > > means one atomic change [1]. The "transactional"
>>>> MetaStore
>>>> > > > > impl.
>>>> > > > > > is
>>>> > > > > > > > > but a
>>>> > > > > > > > > > > sub-case of that. It cannot alter the high-level
>>>> contract.
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > We could add SPI methods having multiple object
>>>> parameters
>>>> > > to
>>>> > > > > > > represent
>>>> > > > > > > > > > > grouped changes, but I am not sure it will be a
>>>> sound
>>>> > > design.
>>>> > > > > > This
>>>> > > > > > > will
>>>> > > > > > > > > > > bloat the interface surfaces and require extra
>>>> impl. effort
>>>> > > > for
>>>> > > > > > > each
>>>> > > > > > > > > > > backend type. More importantly, adding multi-arg
>>>> change
>>>> > > > methods
>>>> > > > > > > still
>>>> > > > > > > > > > won't
>>>> > > > > > > > > > > address the problem of reads being consistent with
>>>> writes,
>>>> > > > > > because
>>>> > > > > > > each
>>>> > > > > > > > > > > method call will still be independent regarding the
>>>> data
>>>> > > > stored
>>>> > > > > > in
>>>> > > > > > > the
>>>> > > > > > > > > > > database.
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > I tend to think we need to introduce a "change set"
>>>> or
>>>> > > > "atomic
>>>> > > > > > > batch"
>>>> > > > > > > > > > > concept to core Persistence and associate each REST
>>>> API
>>>> > > > request
>>>> > > > > > > with
>>>> > > > > > > > > one
>>>> > > > > > > > > > > such change set, which will be committed (or rolled
>>>> back)
>>>> > > at
>>>> > > > > the
>>>> > > > > > > end of
>>>> > > > > > > > > > the
>>>> > > > > > > > > > > request. I believe Ayush mentioned a similar
>>>> concept in PR
>>>> > > > 4939
>>>> > > > > > > [2]. In
>>>> > > > > > > > > > > JDBC each change set will naturally be associated
>>>> with an
>>>> > > > RDBMS
>>>> > > > > > > > > > > transaction. In NoSQL persistence, each atomic
>>>> change set
>>>> > > > will
>>>> > > > > be
>>>> > > > > > > > > > > associated with one CAS operation on the underlying
>>>> > > database.
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > [1]
>>>> > > > > > >
>>>> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > [2]
>>>> > > > > > >
>>>> https://github.com/apache/polaris/pull/4939#discussion_r3575719158
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > Cheers,
>>>> > > > > > > > > > > Dmitri.
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
>>>> > > > flyrain000@gmail.com
>>>> > > > > >
>>>> > > > > > > wrote:
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > > Thanks for raising this, Dmitri.
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > These are valid concerns, and they were already
>>>> > > recognized
>>>> > > > > when
>>>> > > > > > > we
>>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At that
>>>> time, we
>>>> > > > > chose
>>>> > > > > > > to use
>>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC due
>>>> to the
>>>> > > > > > > simplicity. I
>>>> > > > > > > > > > > think
>>>> > > > > > > > > > > > most of the issues mentioned here can already be
>>>> > > addressed
>>>> > > > by
>>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl.
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > For example, rename is already wrapped in a
>>>> transaction
>>>> > > in
>>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. Similarly,
>>>> catalog
>>>> > > > > > > creation,
>>>> > > > > > > > > > which
>>>> > > > > > > > > > > > involves reading and creating multiple objects,
>>>> is also
>>>> > > > > > executed
>>>> > > > > > > > > > within a
>>>> > > > > > > > > > > > transaction [2].
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > I see two possible directions:
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > 1.
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager
>>>> together with
>>>> > > the
>>>> > > > > > > > > persistence
>>>> > > > > > > > > > > > backends (such as JDBC) to provide the required
>>>> > > > > consistency
>>>> > > > > > > > > > guarantees
>>>> > > > > > > > > > > > for
>>>> > > > > > > > > > > > specific operations, similar to what
>>>> > > > > > > > > > TransactionalMetaStoreManagerImpl
>>>> > > > > > > > > > > > does.
>>>> > > > > > > > > > > > 2.
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > Migrate the persistence backends (such as
>>>> JDBC) to use
>>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl directly. We
>>>> may
>>>> > > have
>>>> > > > to
>>>> > > > > > > deal
>>>> > > > > > > > > with
>>>> > > > > > > > > > > > transactional semantic mismatches across
>>>> different
>>>> > > > > > persistence
>>>> > > > > > > > > > > backends.
>>>> > > > > > > > > > > > For example, we would likely avoid using JDBC's
>>>> > > > > > > > > > `runWithinTransaction`
>>>> > > > > > > > > > > > for
>>>> > > > > > > > > > > > single row updates, which adds additional
>>>> overhead and
>>>> > > > > > > complexity
>>>> > > > > > > > > > > > without
>>>> > > > > > > > > > > > benefits.
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > References:
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > 1.
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
>>>> > > > > > > > > > > > 2.
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > Yufei
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri
>>>> Bourlatchkov <
>>>> > > > > > > > > dimas@apache.org>
>>>> > > > > > > > > > > > wrote:
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > > Hi all,
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a couple
>>>> of
>>>> > > > > > interesting
>>>> > > > > > > PRs:
>>>> > > > > > > > > > > > > [4939], [5035].
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > It looks like people are starting to encounter
>>>> > > > consistency
>>>> > > > > > > issues
>>>> > > > > > > > > in
>>>> > > > > > > > > > > > > JDBC persistence.
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > The PRs provide valuable insight into the
>>>> underlying
>>>> > > > > issues.
>>>> > > > > > > They
>>>> > > > > > > > > > offer
>>>> > > > > > > > > > > > > incremental fixes that can work. However, I
>>>> believe it
>>>> > > is
>>>> > > > > > time
>>>> > > > > > > for
>>>> > > > > > > > > > the
>>>> > > > > > > > > > > > > Polaris community to review and improve this
>>>> area of
>>>> > > the
>>>> > > > > > > codebase
>>>> > > > > > > > > > > > > holistically.
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > By this, I mean finding a solution that can be
>>>> applied
>>>> > > to
>>>> > > > > all
>>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC, NoSQL)
>>>> and
>>>> > > > addresses
>>>> > > > > > > these
>>>> > > > > > > > > > > > > aspects:
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > * Supporting concurrent and consistent changes
>>>> where
>>>> > > the
>>>> > > > > > > service
>>>> > > > > > > > > > reads
>>>> > > > > > > > > > > > > and validates current catalog state, then
>>>> commits a
>>>> > > > > change
>>>> > > > > > > (e.g.
>>>> > > > > > > > > > > > > name clashes during renames).
>>>> > > > > > > > > > > > > * Supporting consistent but independent changes
>>>> to RBAC
>>>> > > > > > grants
>>>> > > > > > > and
>>>> > > > > > > > > > > > > MetaStore entities. This independence is
>>>> needed to
>>>> > > > > support
>>>> > > > > > > > > > > > > external authorizers like OPA and Ranger.
>>>> > > > > > > > > > > > > * Supporting atomic changes across multiple
>>>> similar
>>>> > > > > entities.
>>>> > > > > > > > > > > > > * Supporting authorization-based filtering of
>>>> list
>>>> > > > > operations
>>>> > > > > > > (cf.
>>>> > > > > > > > > > > > > [4831]).
>>>> > > > > > > > > > > > > * Supporting credential-vending decisions that
>>>> are
>>>> > > rooted
>>>> > > > > in
>>>> > > > > > > the
>>>> > > > > > > > > > > > > exact state of the catalog.
>>>> > > > > > > > > > > > > * Supporting server-side retries for transient
>>>> > > > persistence
>>>> > > > > > > failures
>>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures).
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > Please share your comments and ideas.
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > [4831]
>>>> https://github.com/apache/polaris/pull/4831
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > [4939]
>>>> https://github.com/apache/polaris/pull/4939
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > [5035]
>>>> https://github.com/apache/polaris/pull/5035
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > Thanks,
>>>> > > > > > > > > > > > > Dmitri
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>>
>>>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi All,
Here's a recap of the Community Sync discussion as I remember it. I'm sure
my recollection is not complete, so please add your thoughts to this thread.
* Polaris traditionally relies on entity version number to ensure
consistency of changes.
This works well for internal RBAC grants, which also have version numbers
cross-referenced at the Persistence layer.
I do not think this works in more general cases like validating locations
wrt catalog-level storage configuration.
Another difficult case involves location overlaps between concurrent table
creations.
* We talked about how Persistence calls related to RDBMS transactions (in
the JDBC persistence case).
There were some concerns raised about whether we need to bind Persistence
API to transactions explicitly.
I think it is a valid concern. At the same time, the Persistence API should
be clear about consistency guarantees across all backend implementations. I
think this needs more attention now that the matter of milti-entioty
changes became prominent in [5035].
I personally believe that RDBMS-based Persistence in Polaris has to use
Serializable transaction isolation. Consequently, the core code needs to
provide mechanisms to allow the Persistence implementation to properly
connect reads and writes from API requests to JDBC transactions... whether
we use "transation" as an explicit terms in the SPI or not.
Note that currently, the same API request performs reads and writes in
_separate_ RDBMS transactions (even separate connections).
* We did not discuss Persistence SPI implications in the call, but
currently each Persistence call is assumed to be one distinct and atomic
change (please correct me if I'm wrong).
Consequently, supporting new use cases involves introducing new SPI methods
with many parameters. This can be seen in PR [5035]. This tends to
overcomplicate the SPI.
Ideally, I think the SPI should be simplified to deal with multi-entity and
single-entity changes in a coherent manner to avoid any ambiguity about
which method the caller should use.
[5035] https://github.com/apache/polaris/pull/5035
Cheers,
Dmitri.
On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Heads up: This discussion is on the Community Sync call agenda for Sept 17.
>
> Interested parties, please plan to attend, if possible :)
>
> Cheers,
> Dmitri.
>
> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
>> Hi JB,
>>
>> I've added an agenda item to the next community sync call for this (Sept
>> 17, 2026).
>>
>> Re: key points: did you mean a real GH discussion or dev email (this
>> thread)? Just double checking :) I'm fine with either approach.
>>
>> Cheers,
>> Dmitri.
>>
>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Hi Dmitri,
>>>
>>> I agree on the need to align on these points, though I'm not entirely
>>> sure a dedicated meeting is necessary. Let's start by using some time
>>> during the next community meeting to discuss it.
>>>
>>> In the meantime, what if we outline the key points in a GitHub
>>> Discussion first? We can then use that as a reference during the
>>> meeting.
>>>
>>> Regards,
>>> JB
>>>
>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <di...@apache.org>
>>> wrote:
>>> >
>>> > Hi All,
>>> >
>>> > My impression from this thread is that it might be time for a call to
>>> > discuss all the related issues and try to arrive at a shared
>>> implementation
>>> > plan.
>>> >
>>> > I know meetings are not ideal, but in this case it might be beneficial
>>> as a
>>> > means for achieving a common understanding of the set of problems and
>>> > priorities related to this thread. A dedicated metrics meeting worked
>>> well
>>> > from my POV.
>>> >
>>> > Allocating a time slot in the next community meeting might be an
>>> option,
>>> > although I think we might need the full hour in this case.
>>> >
>>> > WDYT?
>>> >
>>> > Thanks,
>>> > Dmitri.
>>> >
>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <prithvisivasankar@gmail.com
>>> >
>>> > wrote:
>>> >
>>> > > Hi all,
>>> > >
>>> > > I rewrote https://github.com/apache/polaris/pull/5035, following the
>>> > > discussion to focus only on the SPI foundation.
>>> > >
>>> > > What is in the PR now:
>>> > >
>>> > > - A written manager-level consistency contract in
>>> > > site/content/in-dev/unreleased/persistence-consistency.md.
>>> > > - CAS-aware EntityMutation / GrantMutation records.
>>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet, with
>>> backends
>>> > > opting in via supportsAtomicMixedCommit().
>>> > > - commitChangeSet implementations for the in-memory TreeMap
>>> backend and
>>> > > JDBC.
>>> > > - AtomicOperationMetaStoreManager and
>>> TransactionalMetaStoreManagerImpl
>>> > > grant/revoke paths now build one change set per operation,
>>> falling back
>>> > > to
>>> > > individual operations when the backend does not support mixed
>>> commits.
>>> > > - Tests for the change-set API and TreeMap atomic commits.
>>> > >
>>> > > I intentionally left out:
>>> > >
>>> > > - Entity deletes in MetaStoreChangeSet.
>>> > > - Refactoring createCatalog, dropEntity, and renameEntity to use
>>> change
>>> > > sets.
>>> > > - NoSQL support for commitChangeSet.
>>> > >
>>> > > I am treating this as Phase 1 so the contract and the SPI primitive
>>> can be
>>> > > reviewed before the larger operation migrations land.
>>> > >
>>> > > Could you please take a look? In particular, I would like to know
>>> whether
>>> > > the contract captures the consensus so far, or if it needs to
>>> address retry
>>> > > / external-work coordination before we merge this foundation.
>>> > >
>>> > > Thanks,
>>> > > Prithvi S
>>> > >
>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <
>>> dimas@apache.org>
>>> > > wrote:
>>> > >
>>> > > > Hi Robert,
>>> > > >
>>> > > > I agree that the client-visible operation (e.g. REST API request)
>>> > > consists
>>> > > > of many distinct phases. The database / persistence changes are
>>> just one
>>> > > of
>>> > > > these phases. STS (as an example) is another. My point about
>>> transactions
>>> > > > in JDBC Persistence applies to the former (database changes) phase
>>> - one
>>> > > > retry attempt there should ideally involve exactly one transaction.
>>> > > >
>>> > > > We certainly need logic inside Polaris Servers to to coordinate
>>> requests
>>> > > to
>>> > > > various external systems depending on outcomes from previous
>>> request
>>> > > > processing phases.
>>> > > >
>>> > > > I also agree that validation based on database state should be
>>> repeated
>>> > > > from scratch if we retry the database changes.
>>> > > >
>>> > > > However, internal RBAC authorization naturally has to happen
>>> inside the
>>> > > > database update phase (read-check-write). IIRC, that is part of the
>>> > > > implicit consistency guarantees, which were discussed during early
>>> > > project
>>> > > > months (but never got written down in full clarity,
>>> unfortunately). If
>>> > > the
>>> > > > Authorization check is inside that phase for internal RBAC, I
>>> suppose it
>>> > > > will be there for other Authorizers too and that can involve
>>> external
>>> > > > systems (e.g. OPA / Ranger). We can certainly expect authorization
>>> to be
>>> > > > efficient, but I'm not sure how quick it can be in practice.
>>> There's
>>> > > > certainly potential for delays in some (perhaps infrequent) cases.
>>> > > >
>>> > > > Cheers,
>>> > > > Dmitri.
>>> > > >
>>> > > >
>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <sn...@snazy.de>
>>> wrote:
>>> > > >
>>> > > > > Hi Dmitri,
>>> > > > >
>>> > > > > I agree that one logical operation needs a consistent view of
>>> the whole
>>> > > > > backend state.
>>> > > > > What worries me is treating one long JDBC transaction as the
>>> boundary
>>> > > of
>>> > > > > that operation and then retrying the complete request if the
>>> commit
>>> > > > fails.
>>> > > > >
>>> > > > > The request can also write metadata to object storage or call
>>> STS.
>>> > > > > Each system can leave us with an outcome whose certainty we
>>> cannot
>>> > > know.
>>> > > > > The database may have committed before the connection was lost,
>>> an
>>> > > > > object-store write may have succeeded before a timeout, or STS
>>> may have
>>> > > > > issued credentials before its response was lost.
>>> > > > > A database rollback cannot undo any of those effects, and
>>> retrying the
>>> > > > > whole request may repeat them.
>>> > > > >
>>> > > > > So I think the contract has to separate the overall operation
>>> from each
>>> > > > > attempt to commit it.
>>> > > > > Each database or backend commit attempt should stay short.
>>> > > > > When we retry, validation and authorization need to use the
>>> state for
>>> > > > that
>>> > > > > new attempt, and we need clear rules for which external work can
>>> be
>>> > > > reused,
>>> > > > > repeated, cleaned up, or reconciled after an uncertain outcome.
>>> > > > >
>>> > > > > Some contextual state could be useful for carrying that state.
>>> > > > > But the JDBC connection and transaction should be an
>>> implementation
>>> > > > detail
>>> > > > > inside it, not the boundary of the complete operation.
>>> > > > >
>>> > > > > Cheers,
>>> > > > > Robert
>>> > > > >
>>> > > > >
>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <
>>> dimas@apache.org
>>> > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Hi All,
>>> > > > > >
>>> > > > > > The idea of a backend-agnostic change-set primitive sounds
>>> good.
>>> > > > > >
>>> > > > > > However, I am not sure it is sufficient for all the use cases
>>> we
>>> > > > touched
>>> > > > > > here. More specifically, the state read by validation code is
>>> not
>>> > > > > > necessarily reflected in the change set:
>>> > > > > >
>>> > > > > > * Unchanged entities might be considered by validation, but
>>> changed
>>> > > in
>>> > > > a
>>> > > > > > parallel request.
>>> > > > > > * Even for changed entities the state read by validation may
>>> differ
>>> > > > from
>>> > > > > > the state being altered. Some validation code talks to the
>>> MetaStore
>>> > > > > > directly, outside of the data produced by the Resolver.
>>> > > > > >
>>> > > > > > I do not think Polaris offers any explicit mechanisms (ATM) to
>>> ensure
>>> > > > > > consistency between these reads and subsequent writes.
>>> > > > > >
>>> > > > > > As far as JDBC goes, running an overarching Transaction across
>>> all
>>> > > > reads
>>> > > > > > and writes in the same request, at the SERIALIZABLE isolation
>>> level
>>> > > in
>>> > > > > the
>>> > > > > > backing RDBMS could solve the problem, I think.
>>> > > > > >
>>> > > > > > Indeed, such a transaction might be long to accommodate calls
>>> made by
>>> > > > > > Polaris to external storage, etc. However, is that a problem?
>>> I think
>>> > > > the
>>> > > > > > alternative is for the JDBC Persistence impl. to "manually"
>>> track all
>>> > > > > reads
>>> > > > > > under the same request and redo them in the small transaction
>>> that
>>> > > > > persists
>>> > > > > > the writes. This will also add RDBMS overhead and require
>>> complex
>>> > > code
>>> > > > in
>>> > > > > > Polaris to handle the data properly.
>>> > > > > >
>>> > > > > > Tracking the request-wide transaction can be done only for JDBC
>>> > > without
>>> > > > > > leaking "transaction" concepts to the NoSQL Persistence, I
>>> think.
>>> > > NoSQL
>>> > > > > > will use other mechanisms to ensure read/write consistency.
>>> > > > > >
>>> > > > > > Connecting to Robert's email (a parallel branch in this
>>> discussion),
>>> > > > I'd
>>> > > > > > like to propose this approach:
>>> > > > > >
>>> > > > > > * Each request establishes a "Data Context"
>>> > > > > > - In JDBC the Data Context corresponds to a JDBC Connection
>>> + Tx
>>> > > > > > - In NoSQL the Data Context tracks one or more reference
>>> hashes
>>> > > > > > * All Persistence access in the same request goes through the
>>> same
>>> > > Data
>>> > > > > > Context
>>> > > > > > * All Persistence changes are committed once at the end of the
>>> > > request
>>> > > > > > - Not all changes have to be globally atomic. For example,
>>> changes
>>> > > in
>>> > > > > > Catalogs A and B do not have to be atomic with respect to each
>>> other.
>>> > > > We
>>> > > > > > can go deeper into this later. This is relevant to NoSQL.
>>> > > > > > - Transactional backends like JDBC can, of course, choose to
>>> make
>>> > > all
>>> > > > > > changes globally atomic.
>>> > > > > > * A commit can fail in two main ways:
>>> > > > > > - A retriable failure like an optimistic lock error or Tx
>>> > > > > serializability
>>> > > > > > error
>>> > > > > > - A non-triable logical error (e.g. entity not found)
>>> > > > > > * On a retriable error the whole request is re-attempted (as if
>>> > > > > resubmitted
>>> > > > > > by a client) a few times (configurable timeout).
>>> > > > > >
>>> > > > > > WDYT?
>>> > > > > >
>>> > > > > > Cheers,
>>> > > > > > Dmitri.
>>> > > > > >
>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré <
>>> > > jb@nanthrax.net
>>> > > > >
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > Hi all
>>> > > > > > >
>>> > > > > > > I think Robert has a good point.
>>> > > > > > >
>>> > > > > > > If the atomicity guarantee lives only in BasePersistence,
>>> then the
>>> > > > > > > manager contract can't tell a caller whether the state it
>>> read for
>>> > > > > > > validation/authorization/credential-vending is the same
>>> state that
>>> > > > > > > eventually commits. That's the actual bug class behind the
>>> JDBC
>>> > > > > > > symptoms (not any single operation being non-atomic, but the
>>> > > contract
>>> > > > > > > being silent about it. Every operation-specific fix (like
>>> #4939,
>>> > > > #5035
>>> > > > > > > or #5095) re-answers this question locally and it keeps
>>> recurring.
>>> > > So
>>> > > > > > > the deliverable should start with a written consistency
>>> contract at
>>> > > > > > > the manager level.
>>> > > > > > >
>>> > > > > > > I'm not sure migrating everyone to
>>> > > TransactionalMetaStoreManagerImpl
>>> > > > > > > is a good idea. It would tie the logical change set to a DB
>>> > > > > > > transaction spanning the REST request. It means:
>>> > > > > > > - it holds a durable transaction open across slow external
>>> work
>>> > > > > > > (credential vending, OPA, Ranger, ...)
>>> > > > > > > - it doesn't map to NoSQL
>>> > > > > > > - It wraps single-row updates in runWiithinTransaction,
>>> which is a
>>> > > > > > overhead
>>> > > > > > >
>>> > > > > > > So, I think the transactional manager isn't a portable
>>> target. It's
>>> > > > > > > "only" one backend's strategy.
>>> > > > > > >
>>> > > > > > > I think Privthi's approach is right. A backend-agnostic
>>> change-set
>>> > > > > > > primitive with a documented fallback is the correct shape.
>>> There is
>>> > > > > > > one caveat: without carrying the original entitiy (not just
>>> the new
>>> > > > > > > one) the change set can't express optimistic concurrency.
>>> > > > > > >
>>> > > > > > > I propose the following multi-steps approach:
>>> > > > > > > 1. We write the manager-level consistency contract as Robert
>>> asked.
>>> > > > It
>>> > > > > > > should include the explicit statement that a logical change
>>> set is
>>> > > > not
>>> > > > > > > a request-scope DB transaction.
>>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency pattern)
>>> > > baseline
>>> > > > > > > first-class in EntityMutation before merging the SPI
>>> > > > > > > 3. Then we refactor createCatalog/dropEntity/renameEntity.
>>> > > > > > >
>>> > > > > > > Thoughts?
>>> > > > > > >
>>> > > > > > > Regards
>>> > > > > > > JB
>>> > > > > > >
>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <snazy@snazy.de
>>> >
>>> > > wrote:
>>> > > > > > > >
>>> > > > > > > > Hi all,
>>> > > > > > > >
>>> > > > > > > > Yufei's clarification about where the atomicity guarantee
>>> is
>>> > > > defined
>>> > > > > > > seems
>>> > > > > > > > important.
>>> > > > > > > > If it is a property of the lower-level BasePersistence
>>> contract
>>> > > > > rather
>>> > > > > > > than
>>> > > > > > > > the general PolarisMetaStoreManager contract, the general
>>> > > contract
>>> > > > > does
>>> > > > > > > not
>>> > > > > > > > tell callers whether the state used for validation,
>>> > > authorization,
>>> > > > or
>>> > > > > > > > credential vending is consistent with the change that
>>> eventually
>>> > > > > > commits.
>>> > > > > > > >
>>> > > > > > > > The current PRs suggest that operation-specific
>>> multi-object
>>> > > > methods
>>> > > > > > can
>>> > > > > > > > fix individual cases, while leaving the same contract
>>> question to
>>> > > > > recur
>>> > > > > > > for
>>> > > > > > > > each new case.
>>> > > > > > > >
>>> > > > > > > > I would also avoid defining a logical change set as a
>>> database
>>> > > > > > > transaction
>>> > > > > > > > around an entire REST request.
>>> > > > > > > > That would tie the contract to the backend and database,
>>> and
>>> > > could
>>> > > > > keep
>>> > > > > > > the
>>> > > > > > > > durable attempt open across slow external work.
>>> > > > > > > >
>>> > > > > > > > So is the choice really between the two current manager
>>> > > > > > implementations,
>>> > > > > > > or
>>> > > > > > > > do we first need to revisit the boundary and guarantees
>>> exposed
>>> > > to
>>> > > > > > their
>>> > > > > > > > callers?
>>> > > > > > > >
>>> > > > > > > > Cheers,
>>> > > > > > > > Robert
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov <
>>> > > > > dimas@apache.org
>>> > > > > > >
>>> > > > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > > > Hi Yufei,
>>> > > > > > > > >
>>> > > > > > > > > Thanks for the link. I stand corrected. The "one atomic
>>> change
>>> > > > per
>>> > > > > > > method"
>>> > > > > > > > > contract as defined in javadoc does apply only to
>>> > > BasePersistence
>>> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates to
>>> > > > > > > BasePersistence).
>>> > > > > > > > >
>>> > > > > > > > > Note that all non-test Persistence implementations in the
>>> > > Polaris
>>> > > > > > > codebase
>>> > > > > > > > > extend those classes (which is probably why I was
>>> confused
>>> > > about
>>> > > > > > > atomicity
>>> > > > > > > > > expectations).
>>> > > > > > > > >
>>> > > > > > > > > However, this creates a gap in the Persistence SPI
>>> > > specification.
>>> > > > > If
>>> > > > > > > > > other PolarisMetaStoreManager implementations do not
>>> have to
>>> > > > comply
>>> > > > > > > with
>>> > > > > > > > > this principle, it will create a conceptual difficulty
>>> at call
>>> > > > > sites.
>>> > > > > > > How
>>> > > > > > > > > can PolarisMetaStoreManager callers reason about
>>> consistency
>>> > > and
>>> > > > > > > durability
>>> > > > > > > > > behaviours in general?
>>> > > > > > > > >
>>> > > > > > > > > I believe we need to address that as part of this
>>> discussion.
>>> > > > > > > > >
>>> > > > > > > > > Cheers,
>>> > > > > > > > > Dmitri.
>>> > > > > > > > >
>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <
>>> flyrain000@gmail.com
>>> > > >
>>> > > > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > > > > The MetaStore SPI is currently defined with the idea
>>> that
>>> > > one
>>> > > > > > > method
>>> > > > > > > > > call
>>> > > > > > > > > > means one atomic change.
>>> > > > > > > > > >
>>> > > > > > > > > > If the "MetaStore SPI" refers to the interface
>>> > > > > > > PolarisMetaStoreManager, I
>>> > > > > > > > > > don't think we've ever state each method to be atomic.
>>> We did
>>> > > > > > clarify
>>> > > > > > > > > > atomicity[1] in the interface BasePersistence though.
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > 1.
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > Yufei
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri Bourlatchkov <
>>> > > > > > > dimas@apache.org>
>>> > > > > > > > > > wrote:
>>> > > > > > > > > >
>>> > > > > > > > > > > Hi Yufei,
>>> > > > > > > > > > >
>>> > > > > > > > > > > I agree that JDBC transactions must be handled more
>>> > > > explicitly.
>>> > > > > > > > > However,
>>> > > > > > > > > > > I'm not sure that simply moving to
>>> > > > > > > TransactionalMetaStoreManagerImpl is
>>> > > > > > > > > > > sufficient.
>>> > > > > > > > > > >
>>> > > > > > > > > > > The MetaStore SPI is currently defined with the idea
>>> that
>>> > > one
>>> > > > > > > method
>>> > > > > > > > > call
>>> > > > > > > > > > > means one atomic change [1]. The "transactional"
>>> MetaStore
>>> > > > > impl.
>>> > > > > > is
>>> > > > > > > > > but a
>>> > > > > > > > > > > sub-case of that. It cannot alter the high-level
>>> contract.
>>> > > > > > > > > > >
>>> > > > > > > > > > > We could add SPI methods having multiple object
>>> parameters
>>> > > to
>>> > > > > > > represent
>>> > > > > > > > > > > grouped changes, but I am not sure it will be a sound
>>> > > design.
>>> > > > > > This
>>> > > > > > > will
>>> > > > > > > > > > > bloat the interface surfaces and require extra impl.
>>> effort
>>> > > > for
>>> > > > > > > each
>>> > > > > > > > > > > backend type. More importantly, adding multi-arg
>>> change
>>> > > > methods
>>> > > > > > > still
>>> > > > > > > > > > won't
>>> > > > > > > > > > > address the problem of reads being consistent with
>>> writes,
>>> > > > > > because
>>> > > > > > > each
>>> > > > > > > > > > > method call will still be independent regarding the
>>> data
>>> > > > stored
>>> > > > > > in
>>> > > > > > > the
>>> > > > > > > > > > > database.
>>> > > > > > > > > > >
>>> > > > > > > > > > > I tend to think we need to introduce a "change set"
>>> or
>>> > > > "atomic
>>> > > > > > > batch"
>>> > > > > > > > > > > concept to core Persistence and associate each REST
>>> API
>>> > > > request
>>> > > > > > > with
>>> > > > > > > > > one
>>> > > > > > > > > > > such change set, which will be committed (or rolled
>>> back)
>>> > > at
>>> > > > > the
>>> > > > > > > end of
>>> > > > > > > > > > the
>>> > > > > > > > > > > request. I believe Ayush mentioned a similar concept
>>> in PR
>>> > > > 4939
>>> > > > > > > [2]. In
>>> > > > > > > > > > > JDBC each change set will naturally be associated
>>> with an
>>> > > > RDBMS
>>> > > > > > > > > > > transaction. In NoSQL persistence, each atomic
>>> change set
>>> > > > will
>>> > > > > be
>>> > > > > > > > > > > associated with one CAS operation on the underlying
>>> > > database.
>>> > > > > > > > > > >
>>> > > > > > > > > > > [1]
>>> > > > > > >
>>> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
>>> > > > > > > > > > >
>>> > > > > > > > > > > [2]
>>> > > > > > >
>>> https://github.com/apache/polaris/pull/4939#discussion_r3575719158
>>> > > > > > > > > > >
>>> > > > > > > > > > > Cheers,
>>> > > > > > > > > > > Dmitri.
>>> > > > > > > > > > >
>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
>>> > > > flyrain000@gmail.com
>>> > > > > >
>>> > > > > > > wrote:
>>> > > > > > > > > > >
>>> > > > > > > > > > > > Thanks for raising this, Dmitri.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > These are valid concerns, and they were already
>>> > > recognized
>>> > > > > when
>>> > > > > > > we
>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At that
>>> time, we
>>> > > > > chose
>>> > > > > > > to use
>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC due
>>> to the
>>> > > > > > > simplicity. I
>>> > > > > > > > > > > think
>>> > > > > > > > > > > > most of the issues mentioned here can already be
>>> > > addressed
>>> > > > by
>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > For example, rename is already wrapped in a
>>> transaction
>>> > > in
>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. Similarly,
>>> catalog
>>> > > > > > > creation,
>>> > > > > > > > > > which
>>> > > > > > > > > > > > involves reading and creating multiple objects, is
>>> also
>>> > > > > > executed
>>> > > > > > > > > > within a
>>> > > > > > > > > > > > transaction [2].
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > I see two possible directions:
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > 1.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager together
>>> with
>>> > > the
>>> > > > > > > > > persistence
>>> > > > > > > > > > > > backends (such as JDBC) to provide the required
>>> > > > > consistency
>>> > > > > > > > > > guarantees
>>> > > > > > > > > > > > for
>>> > > > > > > > > > > > specific operations, similar to what
>>> > > > > > > > > > TransactionalMetaStoreManagerImpl
>>> > > > > > > > > > > > does.
>>> > > > > > > > > > > > 2.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > Migrate the persistence backends (such as JDBC)
>>> to use
>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl directly. We
>>> may
>>> > > have
>>> > > > to
>>> > > > > > > deal
>>> > > > > > > > > with
>>> > > > > > > > > > > > transactional semantic mismatches across
>>> different
>>> > > > > > persistence
>>> > > > > > > > > > > backends.
>>> > > > > > > > > > > > For example, we would likely avoid using JDBC's
>>> > > > > > > > > > `runWithinTransaction`
>>> > > > > > > > > > > > for
>>> > > > > > > > > > > > single row updates, which adds additional
>>> overhead and
>>> > > > > > > complexity
>>> > > > > > > > > > > > without
>>> > > > > > > > > > > > benefits.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > References:
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > 1.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
>>> > > > > > > > > > > > 2.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > Yufei
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri
>>> Bourlatchkov <
>>> > > > > > > > > dimas@apache.org>
>>> > > > > > > > > > > > wrote:
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > > Hi all,
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a couple
>>> of
>>> > > > > > interesting
>>> > > > > > > PRs:
>>> > > > > > > > > > > > > [4939], [5035].
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > It looks like people are starting to encounter
>>> > > > consistency
>>> > > > > > > issues
>>> > > > > > > > > in
>>> > > > > > > > > > > > > JDBC persistence.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > The PRs provide valuable insight into the
>>> underlying
>>> > > > > issues.
>>> > > > > > > They
>>> > > > > > > > > > offer
>>> > > > > > > > > > > > > incremental fixes that can work. However, I
>>> believe it
>>> > > is
>>> > > > > > time
>>> > > > > > > for
>>> > > > > > > > > > the
>>> > > > > > > > > > > > > Polaris community to review and improve this
>>> area of
>>> > > the
>>> > > > > > > codebase
>>> > > > > > > > > > > > > holistically.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > By this, I mean finding a solution that can be
>>> applied
>>> > > to
>>> > > > > all
>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC, NoSQL) and
>>> > > > addresses
>>> > > > > > > these
>>> > > > > > > > > > > > > aspects:
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > * Supporting concurrent and consistent changes
>>> where
>>> > > the
>>> > > > > > > service
>>> > > > > > > > > > reads
>>> > > > > > > > > > > > > and validates current catalog state, then
>>> commits a
>>> > > > > change
>>> > > > > > > (e.g.
>>> > > > > > > > > > > > > name clashes during renames).
>>> > > > > > > > > > > > > * Supporting consistent but independent changes
>>> to RBAC
>>> > > > > > grants
>>> > > > > > > and
>>> > > > > > > > > > > > > MetaStore entities. This independence is
>>> needed to
>>> > > > > support
>>> > > > > > > > > > > > > external authorizers like OPA and Ranger.
>>> > > > > > > > > > > > > * Supporting atomic changes across multiple
>>> similar
>>> > > > > entities.
>>> > > > > > > > > > > > > * Supporting authorization-based filtering of
>>> list
>>> > > > > operations
>>> > > > > > > (cf.
>>> > > > > > > > > > > > > [4831]).
>>> > > > > > > > > > > > > * Supporting credential-vending decisions that
>>> are
>>> > > rooted
>>> > > > > in
>>> > > > > > > the
>>> > > > > > > > > > > > > exact state of the catalog.
>>> > > > > > > > > > > > > * Supporting server-side retries for transient
>>> > > > persistence
>>> > > > > > > failures
>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures).
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > Please share your comments and ideas.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > [4831]
>>> https://github.com/apache/polaris/pull/4831
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > [4939]
>>> https://github.com/apache/polaris/pull/4939
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > [5035]
>>> https://github.com/apache/polaris/pull/5035
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > Thanks,
>>> > > > > > > > > > > > > Dmitri
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>>
>>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Heads up: This discussion is on the Community Sync call agenda for Sept 17.
Interested parties, please plan to attend, if possible :)
Cheers,
Dmitri.
On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <di...@apache.org> wrote:
> Hi JB,
>
> I've added an agenda item to the next community sync call for this (Sept
> 17, 2026).
>
> Re: key points: did you mean a real GH discussion or dev email (this
> thread)? Just double checking :) I'm fine with either approach.
>
> Cheers,
> Dmitri.
>
> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi Dmitri,
>>
>> I agree on the need to align on these points, though I'm not entirely
>> sure a dedicated meeting is necessary. Let's start by using some time
>> during the next community meeting to discuss it.
>>
>> In the meantime, what if we outline the key points in a GitHub
>> Discussion first? We can then use that as a reference during the
>> meeting.
>>
>> Regards,
>> JB
>>
>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <di...@apache.org>
>> wrote:
>> >
>> > Hi All,
>> >
>> > My impression from this thread is that it might be time for a call to
>> > discuss all the related issues and try to arrive at a shared
>> implementation
>> > plan.
>> >
>> > I know meetings are not ideal, but in this case it might be beneficial
>> as a
>> > means for achieving a common understanding of the set of problems and
>> > priorities related to this thread. A dedicated metrics meeting worked
>> well
>> > from my POV.
>> >
>> > Allocating a time slot in the next community meeting might be an option,
>> > although I think we might need the full hour in this case.
>> >
>> > WDYT?
>> >
>> > Thanks,
>> > Dmitri.
>> >
>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <pr...@gmail.com>
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > I rewrote https://github.com/apache/polaris/pull/5035, following the
>> > > discussion to focus only on the SPI foundation.
>> > >
>> > > What is in the PR now:
>> > >
>> > > - A written manager-level consistency contract in
>> > > site/content/in-dev/unreleased/persistence-consistency.md.
>> > > - CAS-aware EntityMutation / GrantMutation records.
>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet, with
>> backends
>> > > opting in via supportsAtomicMixedCommit().
>> > > - commitChangeSet implementations for the in-memory TreeMap
>> backend and
>> > > JDBC.
>> > > - AtomicOperationMetaStoreManager and
>> TransactionalMetaStoreManagerImpl
>> > > grant/revoke paths now build one change set per operation, falling
>> back
>> > > to
>> > > individual operations when the backend does not support mixed
>> commits.
>> > > - Tests for the change-set API and TreeMap atomic commits.
>> > >
>> > > I intentionally left out:
>> > >
>> > > - Entity deletes in MetaStoreChangeSet.
>> > > - Refactoring createCatalog, dropEntity, and renameEntity to use
>> change
>> > > sets.
>> > > - NoSQL support for commitChangeSet.
>> > >
>> > > I am treating this as Phase 1 so the contract and the SPI primitive
>> can be
>> > > reviewed before the larger operation migrations land.
>> > >
>> > > Could you please take a look? In particular, I would like to know
>> whether
>> > > the contract captures the consensus so far, or if it needs to address
>> retry
>> > > / external-work coordination before we merge this foundation.
>> > >
>> > > Thanks,
>> > > Prithvi S
>> > >
>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <dimas@apache.org
>> >
>> > > wrote:
>> > >
>> > > > Hi Robert,
>> > > >
>> > > > I agree that the client-visible operation (e.g. REST API request)
>> > > consists
>> > > > of many distinct phases. The database / persistence changes are
>> just one
>> > > of
>> > > > these phases. STS (as an example) is another. My point about
>> transactions
>> > > > in JDBC Persistence applies to the former (database changes) phase
>> - one
>> > > > retry attempt there should ideally involve exactly one transaction.
>> > > >
>> > > > We certainly need logic inside Polaris Servers to to coordinate
>> requests
>> > > to
>> > > > various external systems depending on outcomes from previous request
>> > > > processing phases.
>> > > >
>> > > > I also agree that validation based on database state should be
>> repeated
>> > > > from scratch if we retry the database changes.
>> > > >
>> > > > However, internal RBAC authorization naturally has to happen inside
>> the
>> > > > database update phase (read-check-write). IIRC, that is part of the
>> > > > implicit consistency guarantees, which were discussed during early
>> > > project
>> > > > months (but never got written down in full clarity, unfortunately).
>> If
>> > > the
>> > > > Authorization check is inside that phase for internal RBAC, I
>> suppose it
>> > > > will be there for other Authorizers too and that can involve
>> external
>> > > > systems (e.g. OPA / Ranger). We can certainly expect authorization
>> to be
>> > > > efficient, but I'm not sure how quick it can be in practice. There's
>> > > > certainly potential for delays in some (perhaps infrequent) cases.
>> > > >
>> > > > Cheers,
>> > > > Dmitri.
>> > > >
>> > > >
>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <sn...@snazy.de>
>> wrote:
>> > > >
>> > > > > Hi Dmitri,
>> > > > >
>> > > > > I agree that one logical operation needs a consistent view of the
>> whole
>> > > > > backend state.
>> > > > > What worries me is treating one long JDBC transaction as the
>> boundary
>> > > of
>> > > > > that operation and then retrying the complete request if the
>> commit
>> > > > fails.
>> > > > >
>> > > > > The request can also write metadata to object storage or call STS.
>> > > > > Each system can leave us with an outcome whose certainty we cannot
>> > > know.
>> > > > > The database may have committed before the connection was lost, an
>> > > > > object-store write may have succeeded before a timeout, or STS
>> may have
>> > > > > issued credentials before its response was lost.
>> > > > > A database rollback cannot undo any of those effects, and
>> retrying the
>> > > > > whole request may repeat them.
>> > > > >
>> > > > > So I think the contract has to separate the overall operation
>> from each
>> > > > > attempt to commit it.
>> > > > > Each database or backend commit attempt should stay short.
>> > > > > When we retry, validation and authorization need to use the state
>> for
>> > > > that
>> > > > > new attempt, and we need clear rules for which external work can
>> be
>> > > > reused,
>> > > > > repeated, cleaned up, or reconciled after an uncertain outcome.
>> > > > >
>> > > > > Some contextual state could be useful for carrying that state.
>> > > > > But the JDBC connection and transaction should be an
>> implementation
>> > > > detail
>> > > > > inside it, not the boundary of the complete operation.
>> > > > >
>> > > > > Cheers,
>> > > > > Robert
>> > > > >
>> > > > >
>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <
>> dimas@apache.org
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi All,
>> > > > > >
>> > > > > > The idea of a backend-agnostic change-set primitive sounds good.
>> > > > > >
>> > > > > > However, I am not sure it is sufficient for all the use cases we
>> > > > touched
>> > > > > > here. More specifically, the state read by validation code is
>> not
>> > > > > > necessarily reflected in the change set:
>> > > > > >
>> > > > > > * Unchanged entities might be considered by validation, but
>> changed
>> > > in
>> > > > a
>> > > > > > parallel request.
>> > > > > > * Even for changed entities the state read by validation may
>> differ
>> > > > from
>> > > > > > the state being altered. Some validation code talks to the
>> MetaStore
>> > > > > > directly, outside of the data produced by the Resolver.
>> > > > > >
>> > > > > > I do not think Polaris offers any explicit mechanisms (ATM) to
>> ensure
>> > > > > > consistency between these reads and subsequent writes.
>> > > > > >
>> > > > > > As far as JDBC goes, running an overarching Transaction across
>> all
>> > > > reads
>> > > > > > and writes in the same request, at the SERIALIZABLE isolation
>> level
>> > > in
>> > > > > the
>> > > > > > backing RDBMS could solve the problem, I think.
>> > > > > >
>> > > > > > Indeed, such a transaction might be long to accommodate calls
>> made by
>> > > > > > Polaris to external storage, etc. However, is that a problem? I
>> think
>> > > > the
>> > > > > > alternative is for the JDBC Persistence impl. to "manually"
>> track all
>> > > > > reads
>> > > > > > under the same request and redo them in the small transaction
>> that
>> > > > > persists
>> > > > > > the writes. This will also add RDBMS overhead and require
>> complex
>> > > code
>> > > > in
>> > > > > > Polaris to handle the data properly.
>> > > > > >
>> > > > > > Tracking the request-wide transaction can be done only for JDBC
>> > > without
>> > > > > > leaking "transaction" concepts to the NoSQL Persistence, I
>> think.
>> > > NoSQL
>> > > > > > will use other mechanisms to ensure read/write consistency.
>> > > > > >
>> > > > > > Connecting to Robert's email (a parallel branch in this
>> discussion),
>> > > > I'd
>> > > > > > like to propose this approach:
>> > > > > >
>> > > > > > * Each request establishes a "Data Context"
>> > > > > > - In JDBC the Data Context corresponds to a JDBC Connection +
>> Tx
>> > > > > > - In NoSQL the Data Context tracks one or more reference
>> hashes
>> > > > > > * All Persistence access in the same request goes through the
>> same
>> > > Data
>> > > > > > Context
>> > > > > > * All Persistence changes are committed once at the end of the
>> > > request
>> > > > > > - Not all changes have to be globally atomic. For example,
>> changes
>> > > in
>> > > > > > Catalogs A and B do not have to be atomic with respect to each
>> other.
>> > > > We
>> > > > > > can go deeper into this later. This is relevant to NoSQL.
>> > > > > > - Transactional backends like JDBC can, of course, choose to
>> make
>> > > all
>> > > > > > changes globally atomic.
>> > > > > > * A commit can fail in two main ways:
>> > > > > > - A retriable failure like an optimistic lock error or Tx
>> > > > > serializability
>> > > > > > error
>> > > > > > - A non-triable logical error (e.g. entity not found)
>> > > > > > * On a retriable error the whole request is re-attempted (as if
>> > > > > resubmitted
>> > > > > > by a client) a few times (configurable timeout).
>> > > > > >
>> > > > > > WDYT?
>> > > > > >
>> > > > > > Cheers,
>> > > > > > Dmitri.
>> > > > > >
>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré <
>> > > jb@nanthrax.net
>> > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi all
>> > > > > > >
>> > > > > > > I think Robert has a good point.
>> > > > > > >
>> > > > > > > If the atomicity guarantee lives only in BasePersistence,
>> then the
>> > > > > > > manager contract can't tell a caller whether the state it
>> read for
>> > > > > > > validation/authorization/credential-vending is the same state
>> that
>> > > > > > > eventually commits. That's the actual bug class behind the
>> JDBC
>> > > > > > > symptoms (not any single operation being non-atomic, but the
>> > > contract
>> > > > > > > being silent about it. Every operation-specific fix (like
>> #4939,
>> > > > #5035
>> > > > > > > or #5095) re-answers this question locally and it keeps
>> recurring.
>> > > So
>> > > > > > > the deliverable should start with a written consistency
>> contract at
>> > > > > > > the manager level.
>> > > > > > >
>> > > > > > > I'm not sure migrating everyone to
>> > > TransactionalMetaStoreManagerImpl
>> > > > > > > is a good idea. It would tie the logical change set to a DB
>> > > > > > > transaction spanning the REST request. It means:
>> > > > > > > - it holds a durable transaction open across slow external
>> work
>> > > > > > > (credential vending, OPA, Ranger, ...)
>> > > > > > > - it doesn't map to NoSQL
>> > > > > > > - It wraps single-row updates in runWiithinTransaction, which
>> is a
>> > > > > > overhead
>> > > > > > >
>> > > > > > > So, I think the transactional manager isn't a portable
>> target. It's
>> > > > > > > "only" one backend's strategy.
>> > > > > > >
>> > > > > > > I think Privthi's approach is right. A backend-agnostic
>> change-set
>> > > > > > > primitive with a documented fallback is the correct shape.
>> There is
>> > > > > > > one caveat: without carrying the original entitiy (not just
>> the new
>> > > > > > > one) the change set can't express optimistic concurrency.
>> > > > > > >
>> > > > > > > I propose the following multi-steps approach:
>> > > > > > > 1. We write the manager-level consistency contract as Robert
>> asked.
>> > > > It
>> > > > > > > should include the explicit statement that a logical change
>> set is
>> > > > not
>> > > > > > > a request-scope DB transaction.
>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency pattern)
>> > > baseline
>> > > > > > > first-class in EntityMutation before merging the SPI
>> > > > > > > 3. Then we refactor createCatalog/dropEntity/renameEntity.
>> > > > > > >
>> > > > > > > Thoughts?
>> > > > > > >
>> > > > > > > Regards
>> > > > > > > JB
>> > > > > > >
>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <sn...@snazy.de>
>> > > wrote:
>> > > > > > > >
>> > > > > > > > Hi all,
>> > > > > > > >
>> > > > > > > > Yufei's clarification about where the atomicity guarantee is
>> > > > defined
>> > > > > > > seems
>> > > > > > > > important.
>> > > > > > > > If it is a property of the lower-level BasePersistence
>> contract
>> > > > > rather
>> > > > > > > than
>> > > > > > > > the general PolarisMetaStoreManager contract, the general
>> > > contract
>> > > > > does
>> > > > > > > not
>> > > > > > > > tell callers whether the state used for validation,
>> > > authorization,
>> > > > or
>> > > > > > > > credential vending is consistent with the change that
>> eventually
>> > > > > > commits.
>> > > > > > > >
>> > > > > > > > The current PRs suggest that operation-specific multi-object
>> > > > methods
>> > > > > > can
>> > > > > > > > fix individual cases, while leaving the same contract
>> question to
>> > > > > recur
>> > > > > > > for
>> > > > > > > > each new case.
>> > > > > > > >
>> > > > > > > > I would also avoid defining a logical change set as a
>> database
>> > > > > > > transaction
>> > > > > > > > around an entire REST request.
>> > > > > > > > That would tie the contract to the backend and database, and
>> > > could
>> > > > > keep
>> > > > > > > the
>> > > > > > > > durable attempt open across slow external work.
>> > > > > > > >
>> > > > > > > > So is the choice really between the two current manager
>> > > > > > implementations,
>> > > > > > > or
>> > > > > > > > do we first need to revisit the boundary and guarantees
>> exposed
>> > > to
>> > > > > > their
>> > > > > > > > callers?
>> > > > > > > >
>> > > > > > > > Cheers,
>> > > > > > > > Robert
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov <
>> > > > > dimas@apache.org
>> > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Yufei,
>> > > > > > > > >
>> > > > > > > > > Thanks for the link. I stand corrected. The "one atomic
>> change
>> > > > per
>> > > > > > > method"
>> > > > > > > > > contract as defined in javadoc does apply only to
>> > > BasePersistence
>> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates to
>> > > > > > > BasePersistence).
>> > > > > > > > >
>> > > > > > > > > Note that all non-test Persistence implementations in the
>> > > Polaris
>> > > > > > > codebase
>> > > > > > > > > extend those classes (which is probably why I was confused
>> > > about
>> > > > > > > atomicity
>> > > > > > > > > expectations).
>> > > > > > > > >
>> > > > > > > > > However, this creates a gap in the Persistence SPI
>> > > specification.
>> > > > > If
>> > > > > > > > > other PolarisMetaStoreManager implementations do not have
>> to
>> > > > comply
>> > > > > > > with
>> > > > > > > > > this principle, it will create a conceptual difficulty at
>> call
>> > > > > sites.
>> > > > > > > How
>> > > > > > > > > can PolarisMetaStoreManager callers reason about
>> consistency
>> > > and
>> > > > > > > durability
>> > > > > > > > > behaviours in general?
>> > > > > > > > >
>> > > > > > > > > I believe we need to address that as part of this
>> discussion.
>> > > > > > > > >
>> > > > > > > > > Cheers,
>> > > > > > > > > Dmitri.
>> > > > > > > > >
>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <
>> flyrain000@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > > The MetaStore SPI is currently defined with the idea
>> that
>> > > one
>> > > > > > > method
>> > > > > > > > > call
>> > > > > > > > > > means one atomic change.
>> > > > > > > > > >
>> > > > > > > > > > If the "MetaStore SPI" refers to the interface
>> > > > > > > PolarisMetaStoreManager, I
>> > > > > > > > > > don't think we've ever state each method to be atomic.
>> We did
>> > > > > > clarify
>> > > > > > > > > > atomicity[1] in the interface BasePersistence though.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > 1.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Yufei
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri Bourlatchkov <
>> > > > > > > dimas@apache.org>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Yufei,
>> > > > > > > > > > >
>> > > > > > > > > > > I agree that JDBC transactions must be handled more
>> > > > explicitly.
>> > > > > > > > > However,
>> > > > > > > > > > > I'm not sure that simply moving to
>> > > > > > > TransactionalMetaStoreManagerImpl is
>> > > > > > > > > > > sufficient.
>> > > > > > > > > > >
>> > > > > > > > > > > The MetaStore SPI is currently defined with the idea
>> that
>> > > one
>> > > > > > > method
>> > > > > > > > > call
>> > > > > > > > > > > means one atomic change [1]. The "transactional"
>> MetaStore
>> > > > > impl.
>> > > > > > is
>> > > > > > > > > but a
>> > > > > > > > > > > sub-case of that. It cannot alter the high-level
>> contract.
>> > > > > > > > > > >
>> > > > > > > > > > > We could add SPI methods having multiple object
>> parameters
>> > > to
>> > > > > > > represent
>> > > > > > > > > > > grouped changes, but I am not sure it will be a sound
>> > > design.
>> > > > > > This
>> > > > > > > will
>> > > > > > > > > > > bloat the interface surfaces and require extra impl.
>> effort
>> > > > for
>> > > > > > > each
>> > > > > > > > > > > backend type. More importantly, adding multi-arg
>> change
>> > > > methods
>> > > > > > > still
>> > > > > > > > > > won't
>> > > > > > > > > > > address the problem of reads being consistent with
>> writes,
>> > > > > > because
>> > > > > > > each
>> > > > > > > > > > > method call will still be independent regarding the
>> data
>> > > > stored
>> > > > > > in
>> > > > > > > the
>> > > > > > > > > > > database.
>> > > > > > > > > > >
>> > > > > > > > > > > I tend to think we need to introduce a "change set" or
>> > > > "atomic
>> > > > > > > batch"
>> > > > > > > > > > > concept to core Persistence and associate each REST
>> API
>> > > > request
>> > > > > > > with
>> > > > > > > > > one
>> > > > > > > > > > > such change set, which will be committed (or rolled
>> back)
>> > > at
>> > > > > the
>> > > > > > > end of
>> > > > > > > > > > the
>> > > > > > > > > > > request. I believe Ayush mentioned a similar concept
>> in PR
>> > > > 4939
>> > > > > > > [2]. In
>> > > > > > > > > > > JDBC each change set will naturally be associated
>> with an
>> > > > RDBMS
>> > > > > > > > > > > transaction. In NoSQL persistence, each atomic change
>> set
>> > > > will
>> > > > > be
>> > > > > > > > > > > associated with one CAS operation on the underlying
>> > > database.
>> > > > > > > > > > >
>> > > > > > > > > > > [1]
>> > > > > > >
>> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
>> > > > > > > > > > >
>> > > > > > > > > > > [2]
>> > > > > > >
>> https://github.com/apache/polaris/pull/4939#discussion_r3575719158
>> > > > > > > > > > >
>> > > > > > > > > > > Cheers,
>> > > > > > > > > > > Dmitri.
>> > > > > > > > > > >
>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
>> > > > flyrain000@gmail.com
>> > > > > >
>> > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Thanks for raising this, Dmitri.
>> > > > > > > > > > > >
>> > > > > > > > > > > > These are valid concerns, and they were already
>> > > recognized
>> > > > > when
>> > > > > > > we
>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At that
>> time, we
>> > > > > chose
>> > > > > > > to use
>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC due to
>> the
>> > > > > > > simplicity. I
>> > > > > > > > > > > think
>> > > > > > > > > > > > most of the issues mentioned here can already be
>> > > addressed
>> > > > by
>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl.
>> > > > > > > > > > > >
>> > > > > > > > > > > > For example, rename is already wrapped in a
>> transaction
>> > > in
>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. Similarly,
>> catalog
>> > > > > > > creation,
>> > > > > > > > > > which
>> > > > > > > > > > > > involves reading and creating multiple objects, is
>> also
>> > > > > > executed
>> > > > > > > > > > within a
>> > > > > > > > > > > > transaction [2].
>> > > > > > > > > > > >
>> > > > > > > > > > > > I see two possible directions:
>> > > > > > > > > > > >
>> > > > > > > > > > > > 1.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager together
>> with
>> > > the
>> > > > > > > > > persistence
>> > > > > > > > > > > > backends (such as JDBC) to provide the required
>> > > > > consistency
>> > > > > > > > > > guarantees
>> > > > > > > > > > > > for
>> > > > > > > > > > > > specific operations, similar to what
>> > > > > > > > > > TransactionalMetaStoreManagerImpl
>> > > > > > > > > > > > does.
>> > > > > > > > > > > > 2.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Migrate the persistence backends (such as JDBC)
>> to use
>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl directly. We
>> may
>> > > have
>> > > > to
>> > > > > > > deal
>> > > > > > > > > with
>> > > > > > > > > > > > transactional semantic mismatches across
>> different
>> > > > > > persistence
>> > > > > > > > > > > backends.
>> > > > > > > > > > > > For example, we would likely avoid using JDBC's
>> > > > > > > > > > `runWithinTransaction`
>> > > > > > > > > > > > for
>> > > > > > > > > > > > single row updates, which adds additional
>> overhead and
>> > > > > > > complexity
>> > > > > > > > > > > > without
>> > > > > > > > > > > > benefits.
>> > > > > > > > > > > >
>> > > > > > > > > > > > References:
>> > > > > > > > > > > >
>> > > > > > > > > > > > 1.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
>> > > > > > > > > > > > 2.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yufei
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri Bourlatchkov
>> <
>> > > > > > > > > dimas@apache.org>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi all,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a couple of
>> > > > > > interesting
>> > > > > > > PRs:
>> > > > > > > > > > > > > [4939], [5035].
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > It looks like people are starting to encounter
>> > > > consistency
>> > > > > > > issues
>> > > > > > > > > in
>> > > > > > > > > > > > > JDBC persistence.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > The PRs provide valuable insight into the
>> underlying
>> > > > > issues.
>> > > > > > > They
>> > > > > > > > > > offer
>> > > > > > > > > > > > > incremental fixes that can work. However, I
>> believe it
>> > > is
>> > > > > > time
>> > > > > > > for
>> > > > > > > > > > the
>> > > > > > > > > > > > > Polaris community to review and improve this area
>> of
>> > > the
>> > > > > > > codebase
>> > > > > > > > > > > > > holistically.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > By this, I mean finding a solution that can be
>> applied
>> > > to
>> > > > > all
>> > > > > > > > > > > > > persistence backends (in-memory, JDBC, NoSQL) and
>> > > > addresses
>> > > > > > > these
>> > > > > > > > > > > > > aspects:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > * Supporting concurrent and consistent changes
>> where
>> > > the
>> > > > > > > service
>> > > > > > > > > > reads
>> > > > > > > > > > > > > and validates current catalog state, then
>> commits a
>> > > > > change
>> > > > > > > (e.g.
>> > > > > > > > > > > > > name clashes during renames).
>> > > > > > > > > > > > > * Supporting consistent but independent changes
>> to RBAC
>> > > > > > grants
>> > > > > > > and
>> > > > > > > > > > > > > MetaStore entities. This independence is needed
>> to
>> > > > > support
>> > > > > > > > > > > > > external authorizers like OPA and Ranger.
>> > > > > > > > > > > > > * Supporting atomic changes across multiple
>> similar
>> > > > > entities.
>> > > > > > > > > > > > > * Supporting authorization-based filtering of list
>> > > > > operations
>> > > > > > > (cf.
>> > > > > > > > > > > > > [4831]).
>> > > > > > > > > > > > > * Supporting credential-vending decisions that are
>> > > rooted
>> > > > > in
>> > > > > > > the
>> > > > > > > > > > > > > exact state of the catalog.
>> > > > > > > > > > > > > * Supporting server-side retries for transient
>> > > > persistence
>> > > > > > > failures
>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures).
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Please share your comments and ideas.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > [4831]
>> https://github.com/apache/polaris/pull/4831
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > [4939]
>> https://github.com/apache/polaris/pull/4939
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > [5035]
>> https://github.com/apache/polaris/pull/5035
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > Dmitri
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>>
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi JB,
I've added an agenda item to the next community sync call for this (Sept
17, 2026).
Re: key points: did you mean a real GH discussion or dev email (this
thread)? Just double checking :) I'm fine with either approach.
Cheers,
Dmitri.
On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Dmitri,
>
> I agree on the need to align on these points, though I'm not entirely
> sure a dedicated meeting is necessary. Let's start by using some time
> during the next community meeting to discuss it.
>
> In the meantime, what if we outline the key points in a GitHub
> Discussion first? We can then use that as a reference during the
> meeting.
>
> Regards,
> JB
>
> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
> >
> > Hi All,
> >
> > My impression from this thread is that it might be time for a call to
> > discuss all the related issues and try to arrive at a shared
> implementation
> > plan.
> >
> > I know meetings are not ideal, but in this case it might be beneficial
> as a
> > means for achieving a common understanding of the set of problems and
> > priorities related to this thread. A dedicated metrics meeting worked
> well
> > from my POV.
> >
> > Allocating a time slot in the next community meeting might be an option,
> > although I think we might need the full hour in this case.
> >
> > WDYT?
> >
> > Thanks,
> > Dmitri.
> >
> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <pr...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I rewrote https://github.com/apache/polaris/pull/5035, following the
> > > discussion to focus only on the SPI foundation.
> > >
> > > What is in the PR now:
> > >
> > > - A written manager-level consistency contract in
> > > site/content/in-dev/unreleased/persistence-consistency.md.
> > > - CAS-aware EntityMutation / GrantMutation records.
> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet, with
> backends
> > > opting in via supportsAtomicMixedCommit().
> > > - commitChangeSet implementations for the in-memory TreeMap backend
> and
> > > JDBC.
> > > - AtomicOperationMetaStoreManager and
> TransactionalMetaStoreManagerImpl
> > > grant/revoke paths now build one change set per operation, falling
> back
> > > to
> > > individual operations when the backend does not support mixed
> commits.
> > > - Tests for the change-set API and TreeMap atomic commits.
> > >
> > > I intentionally left out:
> > >
> > > - Entity deletes in MetaStoreChangeSet.
> > > - Refactoring createCatalog, dropEntity, and renameEntity to use
> change
> > > sets.
> > > - NoSQL support for commitChangeSet.
> > >
> > > I am treating this as Phase 1 so the contract and the SPI primitive
> can be
> > > reviewed before the larger operation migrations land.
> > >
> > > Could you please take a look? In particular, I would like to know
> whether
> > > the contract captures the consensus so far, or if it needs to address
> retry
> > > / external-work coordination before we merge this foundation.
> > >
> > > Thanks,
> > > Prithvi S
> > >
> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Robert,
> > > >
> > > > I agree that the client-visible operation (e.g. REST API request)
> > > consists
> > > > of many distinct phases. The database / persistence changes are just
> one
> > > of
> > > > these phases. STS (as an example) is another. My point about
> transactions
> > > > in JDBC Persistence applies to the former (database changes) phase -
> one
> > > > retry attempt there should ideally involve exactly one transaction.
> > > >
> > > > We certainly need logic inside Polaris Servers to to coordinate
> requests
> > > to
> > > > various external systems depending on outcomes from previous request
> > > > processing phases.
> > > >
> > > > I also agree that validation based on database state should be
> repeated
> > > > from scratch if we retry the database changes.
> > > >
> > > > However, internal RBAC authorization naturally has to happen inside
> the
> > > > database update phase (read-check-write). IIRC, that is part of the
> > > > implicit consistency guarantees, which were discussed during early
> > > project
> > > > months (but never got written down in full clarity, unfortunately).
> If
> > > the
> > > > Authorization check is inside that phase for internal RBAC, I
> suppose it
> > > > will be there for other Authorizers too and that can involve external
> > > > systems (e.g. OPA / Ranger). We can certainly expect authorization
> to be
> > > > efficient, but I'm not sure how quick it can be in practice. There's
> > > > certainly potential for delays in some (perhaps infrequent) cases.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > >
> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <sn...@snazy.de> wrote:
> > > >
> > > > > Hi Dmitri,
> > > > >
> > > > > I agree that one logical operation needs a consistent view of the
> whole
> > > > > backend state.
> > > > > What worries me is treating one long JDBC transaction as the
> boundary
> > > of
> > > > > that operation and then retrying the complete request if the commit
> > > > fails.
> > > > >
> > > > > The request can also write metadata to object storage or call STS.
> > > > > Each system can leave us with an outcome whose certainty we cannot
> > > know.
> > > > > The database may have committed before the connection was lost, an
> > > > > object-store write may have succeeded before a timeout, or STS may
> have
> > > > > issued credentials before its response was lost.
> > > > > A database rollback cannot undo any of those effects, and retrying
> the
> > > > > whole request may repeat them.
> > > > >
> > > > > So I think the contract has to separate the overall operation from
> each
> > > > > attempt to commit it.
> > > > > Each database or backend commit attempt should stay short.
> > > > > When we retry, validation and authorization need to use the state
> for
> > > > that
> > > > > new attempt, and we need clear rules for which external work can be
> > > > reused,
> > > > > repeated, cleaned up, or reconciled after an uncertain outcome.
> > > > >
> > > > > Some contextual state could be useful for carrying that state.
> > > > > But the JDBC connection and transaction should be an implementation
> > > > detail
> > > > > inside it, not the boundary of the complete operation.
> > > > >
> > > > > Cheers,
> > > > > Robert
> > > > >
> > > > >
> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <
> dimas@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > The idea of a backend-agnostic change-set primitive sounds good.
> > > > > >
> > > > > > However, I am not sure it is sufficient for all the use cases we
> > > > touched
> > > > > > here. More specifically, the state read by validation code is not
> > > > > > necessarily reflected in the change set:
> > > > > >
> > > > > > * Unchanged entities might be considered by validation, but
> changed
> > > in
> > > > a
> > > > > > parallel request.
> > > > > > * Even for changed entities the state read by validation may
> differ
> > > > from
> > > > > > the state being altered. Some validation code talks to the
> MetaStore
> > > > > > directly, outside of the data produced by the Resolver.
> > > > > >
> > > > > > I do not think Polaris offers any explicit mechanisms (ATM) to
> ensure
> > > > > > consistency between these reads and subsequent writes.
> > > > > >
> > > > > > As far as JDBC goes, running an overarching Transaction across
> all
> > > > reads
> > > > > > and writes in the same request, at the SERIALIZABLE isolation
> level
> > > in
> > > > > the
> > > > > > backing RDBMS could solve the problem, I think.
> > > > > >
> > > > > > Indeed, such a transaction might be long to accommodate calls
> made by
> > > > > > Polaris to external storage, etc. However, is that a problem? I
> think
> > > > the
> > > > > > alternative is for the JDBC Persistence impl. to "manually"
> track all
> > > > > reads
> > > > > > under the same request and redo them in the small transaction
> that
> > > > > persists
> > > > > > the writes. This will also add RDBMS overhead and require complex
> > > code
> > > > in
> > > > > > Polaris to handle the data properly.
> > > > > >
> > > > > > Tracking the request-wide transaction can be done only for JDBC
> > > without
> > > > > > leaking "transaction" concepts to the NoSQL Persistence, I think.
> > > NoSQL
> > > > > > will use other mechanisms to ensure read/write consistency.
> > > > > >
> > > > > > Connecting to Robert's email (a parallel branch in this
> discussion),
> > > > I'd
> > > > > > like to propose this approach:
> > > > > >
> > > > > > * Each request establishes a "Data Context"
> > > > > > - In JDBC the Data Context corresponds to a JDBC Connection +
> Tx
> > > > > > - In NoSQL the Data Context tracks one or more reference hashes
> > > > > > * All Persistence access in the same request goes through the
> same
> > > Data
> > > > > > Context
> > > > > > * All Persistence changes are committed once at the end of the
> > > request
> > > > > > - Not all changes have to be globally atomic. For example,
> changes
> > > in
> > > > > > Catalogs A and B do not have to be atomic with respect to each
> other.
> > > > We
> > > > > > can go deeper into this later. This is relevant to NoSQL.
> > > > > > - Transactional backends like JDBC can, of course, choose to
> make
> > > all
> > > > > > changes globally atomic.
> > > > > > * A commit can fail in two main ways:
> > > > > > - A retriable failure like an optimistic lock error or Tx
> > > > > serializability
> > > > > > error
> > > > > > - A non-triable logical error (e.g. entity not found)
> > > > > > * On a retriable error the whole request is re-attempted (as if
> > > > > resubmitted
> > > > > > by a client) a few times (configurable timeout).
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré <
> > > jb@nanthrax.net
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all
> > > > > > >
> > > > > > > I think Robert has a good point.
> > > > > > >
> > > > > > > If the atomicity guarantee lives only in BasePersistence, then
> the
> > > > > > > manager contract can't tell a caller whether the state it read
> for
> > > > > > > validation/authorization/credential-vending is the same state
> that
> > > > > > > eventually commits. That's the actual bug class behind the JDBC
> > > > > > > symptoms (not any single operation being non-atomic, but the
> > > contract
> > > > > > > being silent about it. Every operation-specific fix (like
> #4939,
> > > > #5035
> > > > > > > or #5095) re-answers this question locally and it keeps
> recurring.
> > > So
> > > > > > > the deliverable should start with a written consistency
> contract at
> > > > > > > the manager level.
> > > > > > >
> > > > > > > I'm not sure migrating everyone to
> > > TransactionalMetaStoreManagerImpl
> > > > > > > is a good idea. It would tie the logical change set to a DB
> > > > > > > transaction spanning the REST request. It means:
> > > > > > > - it holds a durable transaction open across slow external work
> > > > > > > (credential vending, OPA, Ranger, ...)
> > > > > > > - it doesn't map to NoSQL
> > > > > > > - It wraps single-row updates in runWiithinTransaction, which
> is a
> > > > > > overhead
> > > > > > >
> > > > > > > So, I think the transactional manager isn't a portable target.
> It's
> > > > > > > "only" one backend's strategy.
> > > > > > >
> > > > > > > I think Privthi's approach is right. A backend-agnostic
> change-set
> > > > > > > primitive with a documented fallback is the correct shape.
> There is
> > > > > > > one caveat: without carrying the original entitiy (not just
> the new
> > > > > > > one) the change set can't express optimistic concurrency.
> > > > > > >
> > > > > > > I propose the following multi-steps approach:
> > > > > > > 1. We write the manager-level consistency contract as Robert
> asked.
> > > > It
> > > > > > > should include the explicit statement that a logical change
> set is
> > > > not
> > > > > > > a request-scope DB transaction.
> > > > > > > 2. We make Compare And Swap (optimistic-concurrency pattern)
> > > baseline
> > > > > > > first-class in EntityMutation before merging the SPI
> > > > > > > 3. Then we refactor createCatalog/dropEntity/renameEntity.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Regards
> > > > > > > JB
> > > > > > >
> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <sn...@snazy.de>
> > > wrote:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Yufei's clarification about where the atomicity guarantee is
> > > > defined
> > > > > > > seems
> > > > > > > > important.
> > > > > > > > If it is a property of the lower-level BasePersistence
> contract
> > > > > rather
> > > > > > > than
> > > > > > > > the general PolarisMetaStoreManager contract, the general
> > > contract
> > > > > does
> > > > > > > not
> > > > > > > > tell callers whether the state used for validation,
> > > authorization,
> > > > or
> > > > > > > > credential vending is consistent with the change that
> eventually
> > > > > > commits.
> > > > > > > >
> > > > > > > > The current PRs suggest that operation-specific multi-object
> > > > methods
> > > > > > can
> > > > > > > > fix individual cases, while leaving the same contract
> question to
> > > > > recur
> > > > > > > for
> > > > > > > > each new case.
> > > > > > > >
> > > > > > > > I would also avoid defining a logical change set as a
> database
> > > > > > > transaction
> > > > > > > > around an entire REST request.
> > > > > > > > That would tie the contract to the backend and database, and
> > > could
> > > > > keep
> > > > > > > the
> > > > > > > > durable attempt open across slow external work.
> > > > > > > >
> > > > > > > > So is the choice really between the two current manager
> > > > > > implementations,
> > > > > > > or
> > > > > > > > do we first need to revisit the boundary and guarantees
> exposed
> > > to
> > > > > > their
> > > > > > > > callers?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Robert
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Yufei,
> > > > > > > > >
> > > > > > > > > Thanks for the link. I stand corrected. The "one atomic
> change
> > > > per
> > > > > > > method"
> > > > > > > > > contract as defined in javadoc does apply only to
> > > BasePersistence
> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates to
> > > > > > > BasePersistence).
> > > > > > > > >
> > > > > > > > > Note that all non-test Persistence implementations in the
> > > Polaris
> > > > > > > codebase
> > > > > > > > > extend those classes (which is probably why I was confused
> > > about
> > > > > > > atomicity
> > > > > > > > > expectations).
> > > > > > > > >
> > > > > > > > > However, this creates a gap in the Persistence SPI
> > > specification.
> > > > > If
> > > > > > > > > other PolarisMetaStoreManager implementations do not have
> to
> > > > comply
> > > > > > > with
> > > > > > > > > this principle, it will create a conceptual difficulty at
> call
> > > > > sites.
> > > > > > > How
> > > > > > > > > can PolarisMetaStoreManager callers reason about
> consistency
> > > and
> > > > > > > durability
> > > > > > > > > behaviours in general?
> > > > > > > > >
> > > > > > > > > I believe we need to address that as part of this
> discussion.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <
> flyrain000@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > The MetaStore SPI is currently defined with the idea
> that
> > > one
> > > > > > > method
> > > > > > > > > call
> > > > > > > > > > means one atomic change.
> > > > > > > > > >
> > > > > > > > > > If the "MetaStore SPI" refers to the interface
> > > > > > > PolarisMetaStoreManager, I
> > > > > > > > > > don't think we've ever state each method to be atomic.
> We did
> > > > > > clarify
> > > > > > > > > > atomicity[1] in the interface BasePersistence though.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yufei
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri Bourlatchkov <
> > > > > > > dimas@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Yufei,
> > > > > > > > > > >
> > > > > > > > > > > I agree that JDBC transactions must be handled more
> > > > explicitly.
> > > > > > > > > However,
> > > > > > > > > > > I'm not sure that simply moving to
> > > > > > > TransactionalMetaStoreManagerImpl is
> > > > > > > > > > > sufficient.
> > > > > > > > > > >
> > > > > > > > > > > The MetaStore SPI is currently defined with the idea
> that
> > > one
> > > > > > > method
> > > > > > > > > call
> > > > > > > > > > > means one atomic change [1]. The "transactional"
> MetaStore
> > > > > impl.
> > > > > > is
> > > > > > > > > but a
> > > > > > > > > > > sub-case of that. It cannot alter the high-level
> contract.
> > > > > > > > > > >
> > > > > > > > > > > We could add SPI methods having multiple object
> parameters
> > > to
> > > > > > > represent
> > > > > > > > > > > grouped changes, but I am not sure it will be a sound
> > > design.
> > > > > > This
> > > > > > > will
> > > > > > > > > > > bloat the interface surfaces and require extra impl.
> effort
> > > > for
> > > > > > > each
> > > > > > > > > > > backend type. More importantly, adding multi-arg change
> > > > methods
> > > > > > > still
> > > > > > > > > > won't
> > > > > > > > > > > address the problem of reads being consistent with
> writes,
> > > > > > because
> > > > > > > each
> > > > > > > > > > > method call will still be independent regarding the
> data
> > > > stored
> > > > > > in
> > > > > > > the
> > > > > > > > > > > database.
> > > > > > > > > > >
> > > > > > > > > > > I tend to think we need to introduce a "change set" or
> > > > "atomic
> > > > > > > batch"
> > > > > > > > > > > concept to core Persistence and associate each REST API
> > > > request
> > > > > > > with
> > > > > > > > > one
> > > > > > > > > > > such change set, which will be committed (or rolled
> back)
> > > at
> > > > > the
> > > > > > > end of
> > > > > > > > > > the
> > > > > > > > > > > request. I believe Ayush mentioned a similar concept
> in PR
> > > > 4939
> > > > > > > [2]. In
> > > > > > > > > > > JDBC each change set will naturally be associated with
> an
> > > > RDBMS
> > > > > > > > > > > transaction. In NoSQL persistence, each atomic change
> set
> > > > will
> > > > > be
> > > > > > > > > > > associated with one CAS operation on the underlying
> > > database.
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > >
> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
> > > > > > > > > > >
> > > > > > > > > > > [2]
> > > > > > >
> https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
> > > > flyrain000@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for raising this, Dmitri.
> > > > > > > > > > > >
> > > > > > > > > > > > These are valid concerns, and they were already
> > > recognized
> > > > > when
> > > > > > > we
> > > > > > > > > > > > introduced JDBC persistence to Polaris. At that
> time, we
> > > > > chose
> > > > > > > to use
> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC due to
> the
> > > > > > > simplicity. I
> > > > > > > > > > > think
> > > > > > > > > > > > most of the issues mentioned here can already be
> > > addressed
> > > > by
> > > > > > > > > > > > TransactionalMetaStoreManagerImpl.
> > > > > > > > > > > >
> > > > > > > > > > > > For example, rename is already wrapped in a
> transaction
> > > in
> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. Similarly,
> catalog
> > > > > > > creation,
> > > > > > > > > > which
> > > > > > > > > > > > involves reading and creating multiple objects, is
> also
> > > > > > executed
> > > > > > > > > > within a
> > > > > > > > > > > > transaction [2].
> > > > > > > > > > > >
> > > > > > > > > > > > I see two possible directions:
> > > > > > > > > > > >
> > > > > > > > > > > > 1.
> > > > > > > > > > > >
> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager together
> with
> > > the
> > > > > > > > > persistence
> > > > > > > > > > > > backends (such as JDBC) to provide the required
> > > > > consistency
> > > > > > > > > > guarantees
> > > > > > > > > > > > for
> > > > > > > > > > > > specific operations, similar to what
> > > > > > > > > > TransactionalMetaStoreManagerImpl
> > > > > > > > > > > > does.
> > > > > > > > > > > > 2.
> > > > > > > > > > > >
> > > > > > > > > > > > Migrate the persistence backends (such as JDBC)
> to use
> > > > > > > > > > > > TransactionalMetaStoreManagerImpl directly. We may
> > > have
> > > > to
> > > > > > > deal
> > > > > > > > > with
> > > > > > > > > > > > transactional semantic mismatches across different
> > > > > > persistence
> > > > > > > > > > > backends.
> > > > > > > > > > > > For example, we would likely avoid using JDBC's
> > > > > > > > > > `runWithinTransaction`
> > > > > > > > > > > > for
> > > > > > > > > > > > single row updates, which adds additional
> overhead and
> > > > > > > complexity
> > > > > > > > > > > > without
> > > > > > > > > > > > benefits.
> > > > > > > > > > > >
> > > > > > > > > > > > References:
> > > > > > > > > > > >
> > > > > > > > > > > > 1.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > > > > > > > > > > > 2.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > > > > > > > > > > >
> > > > > > > > > > > > Yufei
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri Bourlatchkov <
> > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ayush and Prithvi recently contributed a couple of
> > > > > > interesting
> > > > > > > PRs:
> > > > > > > > > > > > > [4939], [5035].
> > > > > > > > > > > > >
> > > > > > > > > > > > > It looks like people are starting to encounter
> > > > consistency
> > > > > > > issues
> > > > > > > > > in
> > > > > > > > > > > > > JDBC persistence.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The PRs provide valuable insight into the
> underlying
> > > > > issues.
> > > > > > > They
> > > > > > > > > > offer
> > > > > > > > > > > > > incremental fixes that can work. However, I
> believe it
> > > is
> > > > > > time
> > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > Polaris community to review and improve this area
> of
> > > the
> > > > > > > codebase
> > > > > > > > > > > > > holistically.
> > > > > > > > > > > > >
> > > > > > > > > > > > > By this, I mean finding a solution that can be
> applied
> > > to
> > > > > all
> > > > > > > > > > > > > persistence backends (in-memory, JDBC, NoSQL) and
> > > > addresses
> > > > > > > these
> > > > > > > > > > > > > aspects:
> > > > > > > > > > > > >
> > > > > > > > > > > > > * Supporting concurrent and consistent changes
> where
> > > the
> > > > > > > service
> > > > > > > > > > reads
> > > > > > > > > > > > > and validates current catalog state, then
> commits a
> > > > > change
> > > > > > > (e.g.
> > > > > > > > > > > > > name clashes during renames).
> > > > > > > > > > > > > * Supporting consistent but independent changes to
> RBAC
> > > > > > grants
> > > > > > > and
> > > > > > > > > > > > > MetaStore entities. This independence is needed
> to
> > > > > support
> > > > > > > > > > > > > external authorizers like OPA and Ranger.
> > > > > > > > > > > > > * Supporting atomic changes across multiple similar
> > > > > entities.
> > > > > > > > > > > > > * Supporting authorization-based filtering of list
> > > > > operations
> > > > > > > (cf.
> > > > > > > > > > > > > [4831]).
> > > > > > > > > > > > > * Supporting credential-vending decisions that are
> > > rooted
> > > > > in
> > > > > > > the
> > > > > > > > > > > > > exact state of the catalog.
> > > > > > > > > > > > > * Supporting server-side retries for transient
> > > > persistence
> > > > > > > failures
> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please share your comments and ideas.
> > > > > > > > > > > > >
> > > > > > > > > > > > > [4831] https://github.com/apache/polaris/pull/4831
> > > > > > > > > > > > >
> > > > > > > > > > > > > [4939] https://github.com/apache/polaris/pull/4939
> > > > > > > > > > > > >
> > > > > > > > > > > > > [5035] https://github.com/apache/polaris/pull/5035
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Dmitri
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: [DISCUSS] Consistent multi-object changes in Polaris persistence
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Dmitri,
I agree on the need to align on these points, though I'm not entirely
sure a dedicated meeting is necessary. Let's start by using some time
during the next community meeting to discuss it.
In the meantime, what if we outline the key points in a GitHub
Discussion first? We can then use that as a reference during the
meeting.
Regards,
JB
On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <di...@apache.org> wrote:
>
> Hi All,
>
> My impression from this thread is that it might be time for a call to
> discuss all the related issues and try to arrive at a shared implementation
> plan.
>
> I know meetings are not ideal, but in this case it might be beneficial as a
> means for achieving a common understanding of the set of problems and
> priorities related to this thread. A dedicated metrics meeting worked well
> from my POV.
>
> Allocating a time slot in the next community meeting might be an option,
> although I think we might need the full hour in this case.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <pr...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I rewrote https://github.com/apache/polaris/pull/5035, following the
> > discussion to focus only on the SPI foundation.
> >
> > What is in the PR now:
> >
> > - A written manager-level consistency contract in
> > site/content/in-dev/unreleased/persistence-consistency.md.
> > - CAS-aware EntityMutation / GrantMutation records.
> > - MetaStoreChangeSet and BasePersistence#commitChangeSet, with backends
> > opting in via supportsAtomicMixedCommit().
> > - commitChangeSet implementations for the in-memory TreeMap backend and
> > JDBC.
> > - AtomicOperationMetaStoreManager and TransactionalMetaStoreManagerImpl
> > grant/revoke paths now build one change set per operation, falling back
> > to
> > individual operations when the backend does not support mixed commits.
> > - Tests for the change-set API and TreeMap atomic commits.
> >
> > I intentionally left out:
> >
> > - Entity deletes in MetaStoreChangeSet.
> > - Refactoring createCatalog, dropEntity, and renameEntity to use change
> > sets.
> > - NoSQL support for commitChangeSet.
> >
> > I am treating this as Phase 1 so the contract and the SPI primitive can be
> > reviewed before the larger operation migrations land.
> >
> > Could you please take a look? In particular, I would like to know whether
> > the contract captures the consensus so far, or if it needs to address retry
> > / external-work coordination before we merge this foundation.
> >
> > Thanks,
> > Prithvi S
> >
> > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Robert,
> > >
> > > I agree that the client-visible operation (e.g. REST API request)
> > consists
> > > of many distinct phases. The database / persistence changes are just one
> > of
> > > these phases. STS (as an example) is another. My point about transactions
> > > in JDBC Persistence applies to the former (database changes) phase - one
> > > retry attempt there should ideally involve exactly one transaction.
> > >
> > > We certainly need logic inside Polaris Servers to to coordinate requests
> > to
> > > various external systems depending on outcomes from previous request
> > > processing phases.
> > >
> > > I also agree that validation based on database state should be repeated
> > > from scratch if we retry the database changes.
> > >
> > > However, internal RBAC authorization naturally has to happen inside the
> > > database update phase (read-check-write). IIRC, that is part of the
> > > implicit consistency guarantees, which were discussed during early
> > project
> > > months (but never got written down in full clarity, unfortunately). If
> > the
> > > Authorization check is inside that phase for internal RBAC, I suppose it
> > > will be there for other Authorizers too and that can involve external
> > > systems (e.g. OPA / Ranger). We can certainly expect authorization to be
> > > efficient, but I'm not sure how quick it can be in practice. There's
> > > certainly potential for delays in some (perhaps infrequent) cases.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > >
> > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <sn...@snazy.de> wrote:
> > >
> > > > Hi Dmitri,
> > > >
> > > > I agree that one logical operation needs a consistent view of the whole
> > > > backend state.
> > > > What worries me is treating one long JDBC transaction as the boundary
> > of
> > > > that operation and then retrying the complete request if the commit
> > > fails.
> > > >
> > > > The request can also write metadata to object storage or call STS.
> > > > Each system can leave us with an outcome whose certainty we cannot
> > know.
> > > > The database may have committed before the connection was lost, an
> > > > object-store write may have succeeded before a timeout, or STS may have
> > > > issued credentials before its response was lost.
> > > > A database rollback cannot undo any of those effects, and retrying the
> > > > whole request may repeat them.
> > > >
> > > > So I think the contract has to separate the overall operation from each
> > > > attempt to commit it.
> > > > Each database or backend commit attempt should stay short.
> > > > When we retry, validation and authorization need to use the state for
> > > that
> > > > new attempt, and we need clear rules for which external work can be
> > > reused,
> > > > repeated, cleaned up, or reconciled after an uncertain outcome.
> > > >
> > > > Some contextual state could be useful for carrying that state.
> > > > But the JDBC connection and transaction should be an implementation
> > > detail
> > > > inside it, not the boundary of the complete operation.
> > > >
> > > > Cheers,
> > > > Robert
> > > >
> > > >
> > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <dimas@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > The idea of a backend-agnostic change-set primitive sounds good.
> > > > >
> > > > > However, I am not sure it is sufficient for all the use cases we
> > > touched
> > > > > here. More specifically, the state read by validation code is not
> > > > > necessarily reflected in the change set:
> > > > >
> > > > > * Unchanged entities might be considered by validation, but changed
> > in
> > > a
> > > > > parallel request.
> > > > > * Even for changed entities the state read by validation may differ
> > > from
> > > > > the state being altered. Some validation code talks to the MetaStore
> > > > > directly, outside of the data produced by the Resolver.
> > > > >
> > > > > I do not think Polaris offers any explicit mechanisms (ATM) to ensure
> > > > > consistency between these reads and subsequent writes.
> > > > >
> > > > > As far as JDBC goes, running an overarching Transaction across all
> > > reads
> > > > > and writes in the same request, at the SERIALIZABLE isolation level
> > in
> > > > the
> > > > > backing RDBMS could solve the problem, I think.
> > > > >
> > > > > Indeed, such a transaction might be long to accommodate calls made by
> > > > > Polaris to external storage, etc. However, is that a problem? I think
> > > the
> > > > > alternative is for the JDBC Persistence impl. to "manually" track all
> > > > reads
> > > > > under the same request and redo them in the small transaction that
> > > > persists
> > > > > the writes. This will also add RDBMS overhead and require complex
> > code
> > > in
> > > > > Polaris to handle the data properly.
> > > > >
> > > > > Tracking the request-wide transaction can be done only for JDBC
> > without
> > > > > leaking "transaction" concepts to the NoSQL Persistence, I think.
> > NoSQL
> > > > > will use other mechanisms to ensure read/write consistency.
> > > > >
> > > > > Connecting to Robert's email (a parallel branch in this discussion),
> > > I'd
> > > > > like to propose this approach:
> > > > >
> > > > > * Each request establishes a "Data Context"
> > > > > - In JDBC the Data Context corresponds to a JDBC Connection + Tx
> > > > > - In NoSQL the Data Context tracks one or more reference hashes
> > > > > * All Persistence access in the same request goes through the same
> > Data
> > > > > Context
> > > > > * All Persistence changes are committed once at the end of the
> > request
> > > > > - Not all changes have to be globally atomic. For example, changes
> > in
> > > > > Catalogs A and B do not have to be atomic with respect to each other.
> > > We
> > > > > can go deeper into this later. This is relevant to NoSQL.
> > > > > - Transactional backends like JDBC can, of course, choose to make
> > all
> > > > > changes globally atomic.
> > > > > * A commit can fail in two main ways:
> > > > > - A retriable failure like an optimistic lock error or Tx
> > > > serializability
> > > > > error
> > > > > - A non-triable logical error (e.g. entity not found)
> > > > > * On a retriable error the whole request is re-attempted (as if
> > > > resubmitted
> > > > > by a client) a few times (configurable timeout).
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré <
> > jb@nanthrax.net
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all
> > > > > >
> > > > > > I think Robert has a good point.
> > > > > >
> > > > > > If the atomicity guarantee lives only in BasePersistence, then the
> > > > > > manager contract can't tell a caller whether the state it read for
> > > > > > validation/authorization/credential-vending is the same state that
> > > > > > eventually commits. That's the actual bug class behind the JDBC
> > > > > > symptoms (not any single operation being non-atomic, but the
> > contract
> > > > > > being silent about it. Every operation-specific fix (like #4939,
> > > #5035
> > > > > > or #5095) re-answers this question locally and it keeps recurring.
> > So
> > > > > > the deliverable should start with a written consistency contract at
> > > > > > the manager level.
> > > > > >
> > > > > > I'm not sure migrating everyone to
> > TransactionalMetaStoreManagerImpl
> > > > > > is a good idea. It would tie the logical change set to a DB
> > > > > > transaction spanning the REST request. It means:
> > > > > > - it holds a durable transaction open across slow external work
> > > > > > (credential vending, OPA, Ranger, ...)
> > > > > > - it doesn't map to NoSQL
> > > > > > - It wraps single-row updates in runWiithinTransaction, which is a
> > > > > overhead
> > > > > >
> > > > > > So, I think the transactional manager isn't a portable target. It's
> > > > > > "only" one backend's strategy.
> > > > > >
> > > > > > I think Privthi's approach is right. A backend-agnostic change-set
> > > > > > primitive with a documented fallback is the correct shape. There is
> > > > > > one caveat: without carrying the original entitiy (not just the new
> > > > > > one) the change set can't express optimistic concurrency.
> > > > > >
> > > > > > I propose the following multi-steps approach:
> > > > > > 1. We write the manager-level consistency contract as Robert asked.
> > > It
> > > > > > should include the explicit statement that a logical change set is
> > > not
> > > > > > a request-scope DB transaction.
> > > > > > 2. We make Compare And Swap (optimistic-concurrency pattern)
> > baseline
> > > > > > first-class in EntityMutation before merging the SPI
> > > > > > 3. Then we refactor createCatalog/dropEntity/renameEntity.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <sn...@snazy.de>
> > wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Yufei's clarification about where the atomicity guarantee is
> > > defined
> > > > > > seems
> > > > > > > important.
> > > > > > > If it is a property of the lower-level BasePersistence contract
> > > > rather
> > > > > > than
> > > > > > > the general PolarisMetaStoreManager contract, the general
> > contract
> > > > does
> > > > > > not
> > > > > > > tell callers whether the state used for validation,
> > authorization,
> > > or
> > > > > > > credential vending is consistent with the change that eventually
> > > > > commits.
> > > > > > >
> > > > > > > The current PRs suggest that operation-specific multi-object
> > > methods
> > > > > can
> > > > > > > fix individual cases, while leaving the same contract question to
> > > > recur
> > > > > > for
> > > > > > > each new case.
> > > > > > >
> > > > > > > I would also avoid defining a logical change set as a database
> > > > > > transaction
> > > > > > > around an entire REST request.
> > > > > > > That would tie the contract to the backend and database, and
> > could
> > > > keep
> > > > > > the
> > > > > > > durable attempt open across slow external work.
> > > > > > >
> > > > > > > So is the choice really between the two current manager
> > > > > implementations,
> > > > > > or
> > > > > > > do we first need to revisit the boundary and guarantees exposed
> > to
> > > > > their
> > > > > > > callers?
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Robert
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Yufei,
> > > > > > > >
> > > > > > > > Thanks for the link. I stand corrected. The "one atomic change
> > > per
> > > > > > method"
> > > > > > > > contract as defined in javadoc does apply only to
> > BasePersistence
> > > > > > > > and AtomicOperationMetaStoreManager (which delegates to
> > > > > > BasePersistence).
> > > > > > > >
> > > > > > > > Note that all non-test Persistence implementations in the
> > Polaris
> > > > > > codebase
> > > > > > > > extend those classes (which is probably why I was confused
> > about
> > > > > > atomicity
> > > > > > > > expectations).
> > > > > > > >
> > > > > > > > However, this creates a gap in the Persistence SPI
> > specification.
> > > > If
> > > > > > > > other PolarisMetaStoreManager implementations do not have to
> > > comply
> > > > > > with
> > > > > > > > this principle, it will create a conceptual difficulty at call
> > > > sites.
> > > > > > How
> > > > > > > > can PolarisMetaStoreManager callers reason about consistency
> > and
> > > > > > durability
> > > > > > > > behaviours in general?
> > > > > > > >
> > > > > > > > I believe we need to address that as part of this discussion.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <flyrain000@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > > The MetaStore SPI is currently defined with the idea that
> > one
> > > > > > method
> > > > > > > > call
> > > > > > > > > means one atomic change.
> > > > > > > > >
> > > > > > > > > If the "MetaStore SPI" refers to the interface
> > > > > > PolarisMetaStoreManager, I
> > > > > > > > > don't think we've ever state each method to be atomic. We did
> > > > > clarify
> > > > > > > > > atomicity[1] in the interface BasePersistence though.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 1.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yufei
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Yufei,
> > > > > > > > > >
> > > > > > > > > > I agree that JDBC transactions must be handled more
> > > explicitly.
> > > > > > > > However,
> > > > > > > > > > I'm not sure that simply moving to
> > > > > > TransactionalMetaStoreManagerImpl is
> > > > > > > > > > sufficient.
> > > > > > > > > >
> > > > > > > > > > The MetaStore SPI is currently defined with the idea that
> > one
> > > > > > method
> > > > > > > > call
> > > > > > > > > > means one atomic change [1]. The "transactional" MetaStore
> > > > impl.
> > > > > is
> > > > > > > > but a
> > > > > > > > > > sub-case of that. It cannot alter the high-level contract.
> > > > > > > > > >
> > > > > > > > > > We could add SPI methods having multiple object parameters
> > to
> > > > > > represent
> > > > > > > > > > grouped changes, but I am not sure it will be a sound
> > design.
> > > > > This
> > > > > > will
> > > > > > > > > > bloat the interface surfaces and require extra impl. effort
> > > for
> > > > > > each
> > > > > > > > > > backend type. More importantly, adding multi-arg change
> > > methods
> > > > > > still
> > > > > > > > > won't
> > > > > > > > > > address the problem of reads being consistent with writes,
> > > > > because
> > > > > > each
> > > > > > > > > > method call will still be independent regarding the data
> > > stored
> > > > > in
> > > > > > the
> > > > > > > > > > database.
> > > > > > > > > >
> > > > > > > > > > I tend to think we need to introduce a "change set" or
> > > "atomic
> > > > > > batch"
> > > > > > > > > > concept to core Persistence and associate each REST API
> > > request
> > > > > > with
> > > > > > > > one
> > > > > > > > > > such change set, which will be committed (or rolled back)
> > at
> > > > the
> > > > > > end of
> > > > > > > > > the
> > > > > > > > > > request. I believe Ayush mentioned a similar concept in PR
> > > 4939
> > > > > > [2]. In
> > > > > > > > > > JDBC each change set will naturally be associated with an
> > > RDBMS
> > > > > > > > > > transaction. In NoSQL persistence, each atomic change set
> > > will
> > > > be
> > > > > > > > > > associated with one CAS operation on the underlying
> > database.
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
> > > > > > > > > >
> > > > > > > > > > [2]
> > > > > > https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
> > > flyrain000@gmail.com
> > > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for raising this, Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > These are valid concerns, and they were already
> > recognized
> > > > when
> > > > > > we
> > > > > > > > > > > introduced JDBC persistence to Polaris. At that time, we
> > > > chose
> > > > > > to use
> > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC due to the
> > > > > > simplicity. I
> > > > > > > > > > think
> > > > > > > > > > > most of the issues mentioned here can already be
> > addressed
> > > by
> > > > > > > > > > > TransactionalMetaStoreManagerImpl.
> > > > > > > > > > >
> > > > > > > > > > > For example, rename is already wrapped in a transaction
> > in
> > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. Similarly, catalog
> > > > > > creation,
> > > > > > > > > which
> > > > > > > > > > > involves reading and creating multiple objects, is also
> > > > > executed
> > > > > > > > > within a
> > > > > > > > > > > transaction [2].
> > > > > > > > > > >
> > > > > > > > > > > I see two possible directions:
> > > > > > > > > > >
> > > > > > > > > > > 1.
> > > > > > > > > > >
> > > > > > > > > > > Modify AtomicOperationMetaStoreManager together with
> > the
> > > > > > > > persistence
> > > > > > > > > > > backends (such as JDBC) to provide the required
> > > > consistency
> > > > > > > > > guarantees
> > > > > > > > > > > for
> > > > > > > > > > > specific operations, similar to what
> > > > > > > > > TransactionalMetaStoreManagerImpl
> > > > > > > > > > > does.
> > > > > > > > > > > 2.
> > > > > > > > > > >
> > > > > > > > > > > Migrate the persistence backends (such as JDBC) to use
> > > > > > > > > > > TransactionalMetaStoreManagerImpl directly. We may
> > have
> > > to
> > > > > > deal
> > > > > > > > with
> > > > > > > > > > > transactional semantic mismatches across different
> > > > > persistence
> > > > > > > > > > backends.
> > > > > > > > > > > For example, we would likely avoid using JDBC's
> > > > > > > > > `runWithinTransaction`
> > > > > > > > > > > for
> > > > > > > > > > > single row updates, which adds additional overhead and
> > > > > > complexity
> > > > > > > > > > > without
> > > > > > > > > > > benefits.
> > > > > > > > > > >
> > > > > > > > > > > References:
> > > > > > > > > > >
> > > > > > > > > > > 1.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > > > > > > > > > > 2.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > > > > > > > > > >
> > > > > > > > > > > Yufei
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri Bourlatchkov <
> > > > > > > > dimas@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > Ayush and Prithvi recently contributed a couple of
> > > > > interesting
> > > > > > PRs:
> > > > > > > > > > > > [4939], [5035].
> > > > > > > > > > > >
> > > > > > > > > > > > It looks like people are starting to encounter
> > > consistency
> > > > > > issues
> > > > > > > > in
> > > > > > > > > > > > JDBC persistence.
> > > > > > > > > > > >
> > > > > > > > > > > > The PRs provide valuable insight into the underlying
> > > > issues.
> > > > > > They
> > > > > > > > > offer
> > > > > > > > > > > > incremental fixes that can work. However, I believe it
> > is
> > > > > time
> > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > Polaris community to review and improve this area of
> > the
> > > > > > codebase
> > > > > > > > > > > > holistically.
> > > > > > > > > > > >
> > > > > > > > > > > > By this, I mean finding a solution that can be applied
> > to
> > > > all
> > > > > > > > > > > > persistence backends (in-memory, JDBC, NoSQL) and
> > > addresses
> > > > > > these
> > > > > > > > > > > > aspects:
> > > > > > > > > > > >
> > > > > > > > > > > > * Supporting concurrent and consistent changes where
> > the
> > > > > > service
> > > > > > > > > reads
> > > > > > > > > > > > and validates current catalog state, then commits a
> > > > change
> > > > > > (e.g.
> > > > > > > > > > > > name clashes during renames).
> > > > > > > > > > > > * Supporting consistent but independent changes to RBAC
> > > > > grants
> > > > > > and
> > > > > > > > > > > > MetaStore entities. This independence is needed to
> > > > support
> > > > > > > > > > > > external authorizers like OPA and Ranger.
> > > > > > > > > > > > * Supporting atomic changes across multiple similar
> > > > entities.
> > > > > > > > > > > > * Supporting authorization-based filtering of list
> > > > operations
> > > > > > (cf.
> > > > > > > > > > > > [4831]).
> > > > > > > > > > > > * Supporting credential-vending decisions that are
> > rooted
> > > > in
> > > > > > the
> > > > > > > > > > > > exact state of the catalog.
> > > > > > > > > > > > * Supporting server-side retries for transient
> > > persistence
> > > > > > failures
> > > > > > > > > > > > (e.g. RDBMS Tx serializability failures).
> > > > > > > > > > > >
> > > > > > > > > > > > Please share your comments and ideas.
> > > > > > > > > > > >
> > > > > > > > > > > > [4831] https://github.com/apache/polaris/pull/4831
> > > > > > > > > > > >
> > > > > > > > > > > > [4939] https://github.com/apache/polaris/pull/4939
> > > > > > > > > > > >
> > > > > > > > > > > > [5035] https://github.com/apache/polaris/pull/5035
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Dmitri
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >