You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by Alexandre Dutra <ad...@apache.org> on 2026/07/02 10:46:39 UTC
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Hi Russell,
The existing default metastore shouldn't be viewed merely as a
"relatively simple, self-contained implementation": it has unique
quirks, such as returning result types that no other metastore does.
Choosing the status quo simply to avoid making changes is imho the
worst solution.
Transitioning to the NoSQL metastore as the default remains a net
benefit, even if it is only used by 30% of the user base.
However, I agree with keeping the test suite on JDBC + H2 by default
if that ensures a higher level of confidence in our tests ability to
reproduce real production setups.
Thanks,
Alex
On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
<ru...@gmail.com> wrote:
>
> I agree TreeMap and its associated transactional stack should be deprecated.
> It’s test-only, confusing, and not used by JDBC production paths.
>
> I’m less convinced that NoSQL + InMemory is a clearly superior replacement
> for TreeMap in tests. At the storage layer it’s essentially the same class
> of
> thing: an in-memory, non-durable backend. The difference is that it
> exercises
> a completely different MetaStoreManager/persistence stack
> (NoSqlMetaStoreManager
> vs TransactionalMetaStoreManagerImpl), not the JDBC path most deployments
> use. It
> replaces a relatively simple, self-contained implementation with something
> that is
> part of a much more complicated persistence stack, while remaining
> non-durable
> and test-only at the storage layer.
>
> For the problems originally raised (test-grade defaults, dead production
> path, developer confusion),
> JDBC + H2 seems like an actual fix.
>
>
> On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Alex,
> >
> > I tend to agree that the DataSource discussion related to H2 became a bit
> > convoluted, and it's a lot simpler to use the NoSQL in-memory persistence
> > instead.
> >
> > Cheers,
> > Dmitri.
> >
> > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <ad...@apache.org> wrote:
> >
> > > Hi all,
> > >
> > > Some updates on this:
> > >
> > > I recently started a separate discussion regarding dynamic datasource
> > > activation at runtime [1]. This was intended as a prerequisite for
> > > adopting H2 as the default JDBC driver, as suggested on this thread.
> > >
> > > However, that proposal may end up being rejected.
> > >
> > > Given this situation, we should reconsider our options. If the
> > > PostgreSQL + H2 approach is no longer a viable replacement to the
> > > TreeMapMetaStore, I propose an alternative: let's use the NoSQL
> > > metastore with an InMemory backend.
> > >
> > > My reasoning is as follows:
> > >
> > > 1. It remains a production-ready metastore, so strictly superior to
> > > TreeMapMetaStore.
> > >
> > > 2. The metastore state update logic is the same regardless of the
> > > backend, so we're still close to a real production setup (same as with
> > > an in-memory JDBC driver).
> > >
> > > 3. It doesn't require any external dependencies (no bundled driver).
> > >
> > > I'd also note that while the NoSQL metastore is already included
> > > today, it is not actionable because no backend is available. This goes
> > > against the onboarding UX that we're trying to achieve, as users
> > > interested in Polaris with the NoSQL metastore cannot test this setup
> > > with the official image.
> > >
> > > Therefore, I think NoSQL + InMemory would serve as a more appropriate
> > > default setup for the official Polaris image.
> > >
> > > From what I see, it would boil down to adding one line to
> > > polaris-server or polaris-service:
> > >
> > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > >
> > > What are your thoughts on this?
> > >
> > > Thanks,
> > > Alex
> > >
> > > [1]: https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > >
> > >
> > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra <ad...@apache.org>
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I wanted to give an update on this topic: I have started working on
> > > > this, but I would like to have [4812] merged first, since that's a
> > > > prerequisite.
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > >
> > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> > > > <ru...@gmail.com> wrote:
> > > > >
> > > > > I believe the original idea of TreeMapMetastore was that it looked
> > very
> > > > > similar to FoundationDB from an API perspective, so it served well
> > as
> > > a
> > > > > test system for the original backend being developed at SF.
> > > > >
> > > > > I agree that H2 + JDBC makes sense to me now for the project.
> > > > >
> > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra <ad...@apache.org>
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Thank you for your feedback.
> > > > > >
> > > > > > There seems to be a general agreement on the idea of deprecating
> > > > > > TreeMapMetaStore, coupled with the JDBC + H2 solution for tests.
> > > > > >
> > > > > > In most modules, the tests migration won't pose any serious
> > > > > > challenges, as replacing the metastore is generally just a matter
> > of
> > > > > > changing the configuration, and making sure the realm is properly
> > > > > > bootstrapped.
> > > > > >
> > > > > > There will be, however, a few tricky situations in polaris-core: a
> > > few
> > > > > > tests rely on the TreeMapMetaStore, mostly as a test convenience;
> > but
> > > > > > there is no obvious replacement for it in that module. I am however
> > > > > > confident that we can find a solution for that, either based on
> > > mocks,
> > > > > > or by bringing in a real metastore.
> > > > > >
> > > > > > If no objections are raised, I am going to prepare a PR for this.
> > > > > >
> > > > > > Thanks,
> > > > > > Alex
> > > > > >
> > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Alex,
> > > > > > >
> > > > > > > Thanks for starting this thread.
> > > > > > >
> > > > > > > I also find the TransactionalMetaStoreManagerImpl and related
> > call
> > > paths
> > > > > > > conuising in Apache Polaris code.
> > > > > > >
> > > > > > > I support promoting H2 to the default (in memory) Persistence
> > > backend for
> > > > > > > getting started cases. This should also resolve the old H2
> > > evolution
> > > > > > thread
> > > > > > > [1] by ensuring it is used regularly on the same code paths as
> > > > > > PostgreSQL.
> > > > > > >
> > > > > > > Deprecating TransactionalMetaStoreManagerImpl for removal also
> > > sounds
> > > > > > > reasonable to me. Existing downstream users
> > > > > > > of TransactionalMetaStoreManagerImpl will have time to migrate,
> > or
> > > even
> > > > > > > embed that code (per ASF license) into local builds, during the
> > > > > > deprecation
> > > > > > > phase.
> > > > > > >
> > > > > > > [1]
> > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre Dutra <
> > > adutra@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I am writing to ask the community whether it is OK to deprecate
> > > > > > > > TreeMapMetaStore, as well as all the in-memory metastore
> > manager,
> > > > > > > > metastore manager factory, and persistence types that rely
> > > solely on
> > > > > > > > TreeMapMetaStore.
> > > > > > > >
> > > > > > > > As we know, these components are test-grade only, and not
> > > suitable for
> > > > > > > > production. They trigger a production readiness alert on
> > Polaris
> > > > > > > > startup. It's a considerable amount of code that is virtually
> > > dead in
> > > > > > > > production.
> > > > > > > >
> > > > > > > > It's also confusing for developers. E.g. the "transactional"
> > > metastore
> > > > > > > > is not transactional in the JDBC sense of the term, and thus
> > not
> > > used
> > > > > > > > by JDBC persistence. It also has its quirks: some return
> > > statuses are
> > > > > > > > only returned by that manager.
> > > > > > > >
> > > > > > > > However, these components are used in tests, and I agree that
> > > it's
> > > > > > > > useful to have an in-memory version of the persistence layer
> > for
> > > > > > > > tests. But we have today two alternatives that are imho
> > superior
> > > for
> > > > > > > > tests in polaris-runtime-service:
> > > > > > > >
> > > > > > > > - JDBC persistence with H2 backend. There are already a few
> > tests
> > > > > > > > using this setup.
> > > > > > > > - NoSQL persistence with InMemory backend.
> > > > > > > >
> > > > > > > > Both alternatives test real production-grade persistence code.
> > > > > > > >
> > > > > > > > And finally, TreeMapMetaStore is currently the default runtime
> > > > > > > > persistence in application.properties; and the Helm chart also
> > > > > > > > advertises it as the default. These are not sane defaults,
> > imho.
> > > It's
> > > > > > > > always tricky to provide a good default for datastores, but
> > > since JDBC
> > > > > > > > persistence is included by default in the server image, I think
> > > that
> > > > > > > > including the H2 driver by default could give us a saner
> > default
> > > while
> > > > > > > > keeping the out-of-the-box experience intact (the license is
> > > Category
> > > > > > > > A).
> > > > > > > >
> > > > > > > > Concretely:
> > > > > > > >
> > > > > > > > On the MetaStoreManagerFactory hierarchy, the following
> > > > > > > > implementations are completely in-memory:
> > > > > > > >
> > > > > > > > - InMemoryPolarisMetaStoreManagerFactory: could be removed
> > > > > > > >
> > > > > > > > - InMemoryAtomicOperationMetaStoreManagerFactory: could be
> > > removed
> > > > > > > >
> > > > > > > > - LocalPolarisMetaStoreManagerFactory: is the base class of the
> > > two
> > > > > > > > above; imho it can be removed, but since it's an abstract
> > class,
> > > it
> > > > > > > > may have been extended outside Polaris. But neither JDBC nor
> > > NoSQL use
> > > > > > > > it.
> > > > > > > >
> > > > > > > > On the PolarisMetaStoreManager hierarchy:
> > > > > > > >
> > > > > > > > - TransactionalMetaStoreManagerImpl: is only used by
> > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC and NoSQL do not use
> > > it.
> > > > > > > > Could be removed.
> > > > > > > >
> > > > > > > > - TransactionWorkspaceMetaStoreManager however is a different
> > > beast,
> > > > > > > > in spite of the similar name. It is in use today on the commit
> > > path,
> > > > > > > > so should not be removed.
> > > > > > > >
> > > > > > > > On the BasePersistence hierarchy:
> > > > > > > >
> > > > > > > > - TreeMapTransactionalPersistenceImpl and its TreeMapMetaStore
> > > are
> > > > > > > > only used by InMemoryPolarisMetaStoreManagerFactory, and could
> > be
> > > > > > > > removed;
> > > > > > > >
> > > > > > > > - TransactionalPersistence and
> > AbstractTransactionalPersistence:
> > > these
> > > > > > > > are supertypes of TreeMapTransactionalPersistenceImpl and thus
> > > only
> > > > > > > > used for in-memory. They imo can be removed, but they might
> > have
> > > been
> > > > > > > > extended or implemented outside Polaris.
> > > > > > > >
> > > > > > > > I think it's important to keep Polaris code tidy by removing
> > > unused,
> > > > > > > > unimplementable, or test-grade only components.
> > > > > > > >
> > > > > > > > What are your thoughts?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Alex
> > > > > > > >
> > > > > >
> > >
> >
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Yufei,
I do not have data about H2 driver CVEs, but that's not the point.
The question we need to answer here is: given that the H2 driver is used
only for non-prod "getting started" use cases, do we want to risk
H2-related CVEs in Polaris images that will affect all deployments?
IMHO, based on previous emails in this thread the answer is "no", but I do
not feel too strongly about this. Looking forward to hearing more opinions.
Perhaps I misinterpreted previous emails.
Cheers,
Dmitri.
On Wed, Jul 22, 2026 at 1:09 PM Yufei Gu <fl...@gmail.com> wrote:
> Dmitri, that's a valid concern, but not for the H2 lib only. Every library
> introduced but not-used in production(e.g., test-only) could have CVEs. Do
> we have data showing that the H2 lib tends to have more CVEs than others?
>
> Yufei
>
>
> On Wed, Jul 22, 2026 at 9:30 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Yufei,
> >
> > The specific concerns were raised by Romain and Russell, as far as I can
> > tell.
> >
> > Repeating the links to those messages for convenience: [1], [2].
> >
> > My personal interpretation is that it's not so much about CVEs that exist
> > today, but about something that may be discovered in the future and
> impact
> > the previously released Poalris images with the affected H2 driver
> > versions.
> >
> > From my POV, it's a valid concern given that H2 is not actually meant for
> > production use.
> >
> > WDYT?
> >
> > [1]: https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
> >
> > [2]: https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
> >
> > Cheers,
> > Dmitri.
> >
> > On Tue, Jul 21, 2026 at 6:54 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Could you help me understand the security concern? Every library can
> have
> > > CVEs. Do you mean that the H2 library currently has unresolved CVEs, or
> > > that it has historically had more CVEs than other libraries?
> > >
> > > Yufei
> > >
> > >
> > > On Fri, Jul 17, 2026 at 11:18 AM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > wrote:
> > >
> > > > Hi Yufei,
> > > >
> > > > What is your take on the security concerns associated with including
> > the
> > > H2
> > > > JDBC driver in the standard Polaris images (highlighted by Alex in
> this
> > > > thread on Jul 3) [1]?
> > > >
> > > > [1] https://lists.apache.org/thread/8v42g3rvmt74b9s6ccgy7y8kky2hfbjo
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Fri, Jul 17, 2026 at 2:06 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > >
> > > > > Hi Alex,
> > > > >
> > > > > Thanks for summarizing the discussion and separating the two
> > > questions. I
> > > > > think that helps move us toward consensus.
> > > > >
> > > > > First of all, I agree with the direction, but I'm not sure it's
> > urgent.
> > > > > TreeMapMetaStore isn't perfect, but we already have both JDBC + H2
> > and
> > > > > NoSQL + in-memory in place. That should already satisfy most of
> test
> > > > > coverage.
> > > > >
> > > > > For question 1, I'd prefer Option 1a (JDBC + H2).
> > > > >
> > > > > My main reason is that it exercises the same persistence layer,
> > > > transaction
> > > > > handling, schema management, and configuration that most production
> > > > > deployments use. That gives us a better out of the box experience
> for
> > > > both
> > > > > users and developers, while also increasing confidence that what
> > works
> > > > > locally will behave similarly in production.
> > > > >
> > > > > Do we have to wait for either PR 4812 or 4984? I think bundling the
> > H2
> > > > > driver should just unblock 1a.
> > > > >
> > > > > For question 2, I agree with handling test migration on a case by
> > case
> > > > > basis. Different tests have different goals, and using the most
> > > > appropriate
> > > > > backend for each test seems reasonable.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Fri, Jul 17, 2026 at 10:45 AM Dmitri Bourlatchkov <
> > dimas@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Alex,
> > > > > >
> > > > > > Option 1b certainly makes sense to me. I support going in that
> > > > direction.
> > > > > >
> > > > > > I'd propose opening a PR for that merging it before proceeding to
> > > test
> > > > > code
> > > > > > changes.
> > > > > >
> > > > > > In general, if 1b is acceptable, I'd think H2 should be removed
> [1]
> > > > from
> > > > > > tests and replaced with PostgreSQL or NoSQL InMemory. With that
> in
> > > > mind,
> > > > > > updating tests on a case-by-case basis seems reasonable to me. We
> > can
> > > > > > decide separately in each PR.
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 17, 2026 at 12:59 PM Alexandre Dutra <
> > adutra@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Dmitri, hi all,
> > > > > > >
> > > > > > > Unfortunately I'm not sure your interpretation matches mine, so
> > > let's
> > > > > > > try to wrap up this discussion methodically.
> > > > > > >
> > > > > > > We agreed to deprecate TreeMapMetaStore for removal. But we
> > haven't
> > > > > > > reached consensus on two questions:
> > > > > > >
> > > > > > > 1) What should be the default metastore in the server image?
> > > > > > > 2) How to replace TreeMapMetaStore in tests?
> > > > > > >
> > > > > > > Question 1) is tricky and needs a clear decision. We have only
> > two
> > > > > > > options for a smooth UX (aka, "docker run apache/polaris" just
> > > > works):
> > > > > > >
> > > > > > > - Option 1a: JDBC+H2
> > > > > > > - Option 1b: NoSQL+InMemory
> > > > > > >
> > > > > > > Option 1a is hard because it requires either my work on [4812]
> > > > > > > (runtime activated datasources) or Yufei's work on [4984]
> > > (unmanaged
> > > > > > > datasources). It also *requires* the H2 driver to be bundled by
> > > > > > > default, which some people are not comfortable with. In short,
> I
> > > > don't
> > > > > > > see this option being feasible any time soon.
> > > > > > >
> > > > > > > Therefore, I think option 1b is much easier, and that would be
> my
> > > > > > > preference.
> > > > > > >
> > > > > > > Question 2) is not as urgent, as I'm fairly sure we can find a
> > > > > > > solution as we go. I can see a few options:
> > > > > > > - Option 2a: migrate to JDBC+H2
> > > > > > > - Option 2b: migrate to NoSQL+InMemory
> > > > > > > - Option 2c: decide on a case-by-case basis
> > > > > > >
> > > > > > > My personal preference would be 2c.
> > > > > > >
> > > > > > > Can we try to reach consensus on the two questions please?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Alex
> > > > > > >
> > > > > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > > > > > [4984]: https://github.com/apache/polaris/pull/4984
> > > > > > >
> > > > > > > On Mon, Jul 6, 2026 at 8:29 PM Dmitri Bourlatchkov <
> > > dimas@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > My interpretation of the discussion so far is that:
> > > > > > > >
> > > > > > > > 1) We do want to deprecate the TreeMap MetaStore for removal
> > > > > > > >
> > > > > > > > 2) JDBC + PostgreSQL should be used for testing as well as
> > NoSQL
> > > +
> > > > > > > MongoDB
> > > > > > > > (I believe it is in fact the case in current CI).
> > > > > > > >
> > > > > > > > 3) Instead of the TreeMap MetaStore, we will use the
> in-memory
> > > > NoSQL
> > > > > > > > MetaStore (for lightweight tests)
> > > > > > > >
> > > > > > > > Now, H2 becomes irrevant, IMHO, but removing it is being
> > > discussed
> > > > > > > > separately [1].
> > > > > > > >
> > > > > > > > How does that sound?
> > > > > > > >
> > > > > > > > [1]
> > > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Fri, Jul 3, 2026 at 5:25 AM Alexandre Dutra <
> > > adutra@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I'm confused: security concerns were previously raised here
> > [1]
> > > > and
> > > > > > > > > here [2] about bundling the H2 driver due to its past
> record
> > of
> > > > > > > > > critical CVEs.
> > > > > > > > >
> > > > > > > > > As a result, I assumed the JDBC+H2 option had been
> abandoned.
> > > Has
> > > > > > this
> > > > > > > > > proposal been brought back into consideration?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Alex
> > > > > > > > >
> > > > > > > > > [1]:
> > > > > > https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
> > > > > > > > > [2]:
> > > > > > https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jul 3, 2026 at 8:54 AM Jean-Baptiste Onofré <
> > > > > jb@nanthrax.net
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi
> > > > > > > > > >
> > > > > > > > > > Yeah fully agree and it was my point when I replied on
> the
> > > > thread
> > > > > > > > > > initially.
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > JB
> > > > > > > > > >
> > > > > > > > > > Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <
> > flyrain000@gmail.com
> > > >
> > > > a
> > > > > > > écrit :
> > > > > > > > > >
> > > > > > > > > > > If our goal is to make the default experience closer to
> > > > > > production
> > > > > > > > > > > while removing TreeMap, then JDBC with H2 seems like a
> > more
> > > > > > natural
> > > > > > > > > > > choice. It exercises the same persistence layer, schema
> > > > > > management,
> > > > > > > > > > > and configuration that users will encounter in
> > production,
> > > > > while
> > > > > > > still
> > > > > > > > > > > keeping setup lightweight.
> > > > > > > > > > >
> > > > > > > > > > > Yufei
> > > > > > > > > > >
> > > > > > > > > > > Yufei
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <
> > > > > > adutra@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Russell,
> > > > > > > > > > > >
> > > > > > > > > > > > The existing default metastore shouldn't be viewed
> > merely
> > > > as
> > > > > a
> > > > > > > > > > > > "relatively simple, self-contained implementation":
> it
> > > has
> > > > > > unique
> > > > > > > > > > > > quirks, such as returning result types that no other
> > > > > metastore
> > > > > > > does.
> > > > > > > > > > > >
> > > > > > > > > > > > Choosing the status quo simply to avoid making
> changes
> > is
> > > > > imho
> > > > > > > the
> > > > > > > > > > > > worst solution.
> > > > > > > > > > > >
> > > > > > > > > > > > Transitioning to the NoSQL metastore as the default
> > > > remains a
> > > > > > net
> > > > > > > > > > > > benefit, even if it is only used by 30% of the user
> > base.
> > > > > > > > > > > >
> > > > > > > > > > > > However, I agree with keeping the test suite on JDBC
> +
> > H2
> > > > by
> > > > > > > default
> > > > > > > > > > > > if that ensures a higher level of confidence in our
> > tests
> > > > > > > ability to
> > > > > > > > > > > > reproduce real production setups.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Alex
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
> > > > > > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > I agree TreeMap and its associated transactional
> > stack
> > > > > should
> > > > > > > be
> > > > > > > > > > > deprecated.
> > > > > > > > > > > > > It’s test-only, confusing, and not used by JDBC
> > > > production
> > > > > > > paths.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I’m less convinced that NoSQL + InMemory is a
> clearly
> > > > > > superior
> > > > > > > > > > > replacement
> > > > > > > > > > > > > for TreeMap in tests. At the storage layer it’s
> > > > essentially
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > > class
> > > > > > > > > > > > > of
> > > > > > > > > > > > > thing: an in-memory, non-durable backend. The
> > > difference
> > > > > is
> > > > > > > that
> > > > > > > > > it
> > > > > > > > > > > > > exercises
> > > > > > > > > > > > > a completely different MetaStoreManager/persistence
> > > stack
> > > > > > > > > > > > > (NoSqlMetaStoreManager
> > > > > > > > > > > > > vs TransactionalMetaStoreManagerImpl), not the JDBC
> > > path
> > > > > most
> > > > > > > > > > > deployments
> > > > > > > > > > > > > use. It
> > > > > > > > > > > > > replaces a relatively simple, self-contained
> > > > implementation
> > > > > > > with
> > > > > > > > > > > something
> > > > > > > > > > > > > that is
> > > > > > > > > > > > > part of a much more complicated persistence stack,
> > > while
> > > > > > > remaining
> > > > > > > > > > > > > non-durable
> > > > > > > > > > > > > and test-only at the storage layer.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For the problems originally raised (test-grade
> > > defaults,
> > > > > dead
> > > > > > > > > > > production
> > > > > > > > > > > > > path, developer confusion),
> > > > > > > > > > > > > JDBC + H2 seems like an actual fix.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri
> Bourlatchkov <
> > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I tend to agree that the DataSource discussion
> > > related
> > > > to
> > > > > > H2
> > > > > > > > > became
> > > > > > > > > > > a bit
> > > > > > > > > > > > > > convoluted, and it's a lot simpler to use the
> NoSQL
> > > > > > in-memory
> > > > > > > > > > > persistence
> > > > > > > > > > > > > > instead.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <
> > > > > > > > > adutra@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Some updates on this:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I recently started a separate discussion
> > regarding
> > > > > > dynamic
> > > > > > > > > > > datasource
> > > > > > > > > > > > > > > activation at runtime [1]. This was intended
> as a
> > > > > > > prerequisite
> > > > > > > > > for
> > > > > > > > > > > > > > > adopting H2 as the default JDBC driver, as
> > > suggested
> > > > on
> > > > > > > this
> > > > > > > > > > > thread.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > However, that proposal may end up being
> rejected.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Given this situation, we should reconsider our
> > > > options.
> > > > > > If
> > > > > > > the
> > > > > > > > > > > > > > > PostgreSQL + H2 approach is no longer a viable
> > > > > > replacement
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > > > TreeMapMetaStore, I propose an alternative:
> let's
> > > use
> > > > > the
> > > > > > > NoSQL
> > > > > > > > > > > > > > > metastore with an InMemory backend.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > My reasoning is as follows:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. It remains a production-ready metastore, so
> > > > strictly
> > > > > > > > > superior to
> > > > > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. The metastore state update logic is the same
> > > > > > regardless
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > > > backend, so we're still close to a real
> > production
> > > > > setup
> > > > > > > (same
> > > > > > > > > as
> > > > > > > > > > > with
> > > > > > > > > > > > > > > an in-memory JDBC driver).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3. It doesn't require any external dependencies
> > (no
> > > > > > bundled
> > > > > > > > > > > driver).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'd also note that while the NoSQL metastore is
> > > > already
> > > > > > > > > included
> > > > > > > > > > > > > > > today, it is not actionable because no backend
> is
> > > > > > > available.
> > > > > > > > > This
> > > > > > > > > > > goes
> > > > > > > > > > > > > > > against the onboarding UX that we're trying to
> > > > achieve,
> > > > > > as
> > > > > > > > > users
> > > > > > > > > > > > > > > interested in Polaris with the NoSQL metastore
> > > cannot
> > > > > > test
> > > > > > > this
> > > > > > > > > > > setup
> > > > > > > > > > > > > > > with the official image.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Therefore, I think NoSQL + InMemory would serve
> > as
> > > a
> > > > > more
> > > > > > > > > > > appropriate
> > > > > > > > > > > > > > > default setup for the official Polaris image.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > From what I see, it would boil down to adding
> one
> > > > line
> > > > > to
> > > > > > > > > > > > > > > polaris-server or polaris-service:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What are your thoughts on this?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > [1]:
> > > > > > > > > > >
> > > > > https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre
> Dutra
> > <
> > > > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I wanted to give an update on this topic: I
> > have
> > > > > > started
> > > > > > > > > working
> > > > > > > > > > > on
> > > > > > > > > > > > > > > > this, but I would like to have [4812] merged
> > > first,
> > > > > > since
> > > > > > > > > that's
> > > > > > > > > > > a
> > > > > > > > > > > > > > > > prerequisite.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > [4812]:
> > > > https://github.com/apache/polaris/pull/4812
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell
> Spitzer
> > > > > > > > > > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I believe the original idea of
> > TreeMapMetastore
> > > > was
> > > > > > > that it
> > > > > > > > > > > looked
> > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > similar to FoundationDB from an API
> > > perspective,
> > > > > so
> > > > > > it
> > > > > > > > > served
> > > > > > > > > > > well
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > test system for the original backend being
> > > > > developed
> > > > > > > at SF.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I agree that H2 + JDBC makes sense to me
> now
> > > for
> > > > > the
> > > > > > > > > project.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre
> > > Dutra
> > > > <
> > > > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thank you for your feedback.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > There seems to be a general agreement on
> > the
> > > > idea
> > > > > > of
> > > > > > > > > > > deprecating
> > > > > > > > > > > > > > > > > > TreeMapMetaStore, coupled with the JDBC +
> > H2
> > > > > > > solution for
> > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > In most modules, the tests migration
> won't
> > > pose
> > > > > any
> > > > > > > > > serious
> > > > > > > > > > > > > > > > > > challenges, as replacing the metastore is
> > > > > generally
> > > > > > > just
> > > > > > > > > a
> > > > > > > > > > > matter
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > changing the configuration, and making
> sure
> > > the
> > > > > > > realm is
> > > > > > > > > > > properly
> > > > > > > > > > > > > > > > > > bootstrapped.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > There will be, however, a few tricky
> > > situations
> > > > > in
> > > > > > > > > > > polaris-core: a
> > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > tests rely on the TreeMapMetaStore,
> mostly
> > > as a
> > > > > > test
> > > > > > > > > > > convenience;
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > there is no obvious replacement for it in
> > > that
> > > > > > > module. I
> > > > > > > > > am
> > > > > > > > > > > however
> > > > > > > > > > > > > > > > > > confident that we can find a solution for
> > > that,
> > > > > > > either
> > > > > > > > > based
> > > > > > > > > > > on
> > > > > > > > > > > > > > > mocks,
> > > > > > > > > > > > > > > > > > or by bringing in a real metastore.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > If no objections are raised, I am going
> to
> > > > > prepare
> > > > > > a
> > > > > > > PR
> > > > > > > > > for
> > > > > > > > > > > this.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri
> > > > > > Bourlatchkov <
> > > > > > > > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for starting this thread.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I also find the
> > > > > TransactionalMetaStoreManagerImpl
> > > > > > > and
> > > > > > > > > > > related
> > > > > > > > > > > > > > call
> > > > > > > > > > > > > > > paths
> > > > > > > > > > > > > > > > > > > conuising in Apache Polaris code.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I support promoting H2 to the default
> (in
> > > > > memory)
> > > > > > > > > > > Persistence
> > > > > > > > > > > > > > > backend for
> > > > > > > > > > > > > > > > > > > getting started cases. This should also
> > > > resolve
> > > > > > the
> > > > > > > > > old H2
> > > > > > > > > > > > > > > evolution
> > > > > > > > > > > > > > > > > > thread
> > > > > > > > > > > > > > > > > > > [1] by ensuring it is used regularly on
> > the
> > > > > same
> > > > > > > code
> > > > > > > > > > > paths as
> > > > > > > > > > > > > > > > > > PostgreSQL.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Deprecating
> > > TransactionalMetaStoreManagerImpl
> > > > > for
> > > > > > > > > removal
> > > > > > > > > > > also
> > > > > > > > > > > > > > > sounds
> > > > > > > > > > > > > > > > > > > reasonable to me. Existing downstream
> > users
> > > > > > > > > > > > > > > > > > > of TransactionalMetaStoreManagerImpl
> will
> > > > have
> > > > > > > time to
> > > > > > > > > > > migrate,
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > even
> > > > > > > > > > > > > > > > > > > embed that code (per ASF license) into
> > > local
> > > > > > > builds,
> > > > > > > > > > > during the
> > > > > > > > > > > > > > > > > > deprecation
> > > > > > > > > > > > > > > > > > > phase.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > >
> > > > > > > > >
> > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM
> > Alexandre
> > > > > Dutra
> > > > > > <
> > > > > > > > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I am writing to ask the community
> > whether
> > > > it
> > > > > is
> > > > > > > OK to
> > > > > > > > > > > deprecate
> > > > > > > > > > > > > > > > > > > > TreeMapMetaStore, as well as all the
> > > > > in-memory
> > > > > > > > > metastore
> > > > > > > > > > > > > > manager,
> > > > > > > > > > > > > > > > > > > > metastore manager factory, and
> > > persistence
> > > > > > types
> > > > > > > that
> > > > > > > > > > > rely
> > > > > > > > > > > > > > > solely on
> > > > > > > > > > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > As we know, these components are
> > > test-grade
> > > > > > only,
> > > > > > > > > and not
> > > > > > > > > > > > > > > suitable for
> > > > > > > > > > > > > > > > > > > > production. They trigger a production
> > > > > readiness
> > > > > > > > > alert on
> > > > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > > > > > startup. It's a considerable amount
> of
> > > code
> > > > > > that
> > > > > > > is
> > > > > > > > > > > virtually
> > > > > > > > > > > > > > > dead in
> > > > > > > > > > > > > > > > > > > > production.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It's also confusing for developers.
> > E.g.
> > > > the
> > > > > > > > > > > "transactional"
> > > > > > > > > > > > > > > metastore
> > > > > > > > > > > > > > > > > > > > is not transactional in the JDBC
> sense
> > of
> > > > the
> > > > > > > term,
> > > > > > > > > and
> > > > > > > > > > > thus
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > by JDBC persistence. It also has its
> > > > quirks:
> > > > > > some
> > > > > > > > > return
> > > > > > > > > > > > > > > statuses are
> > > > > > > > > > > > > > > > > > > > only returned by that manager.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > However, these components are used in
> > > > tests,
> > > > > > and
> > > > > > > I
> > > > > > > > > agree
> > > > > > > > > > > that
> > > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > > > > > > > useful to have an in-memory version
> of
> > > the
> > > > > > > > > persistence
> > > > > > > > > > > layer
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > tests. But we have today two
> > alternatives
> > > > > that
> > > > > > > are
> > > > > > > > > imho
> > > > > > > > > > > > > > superior
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > tests in polaris-runtime-service:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > - JDBC persistence with H2 backend.
> > There
> > > > are
> > > > > > > > > already a
> > > > > > > > > > > few
> > > > > > > > > > > > > > tests
> > > > > > > > > > > > > > > > > > > > using this setup.
> > > > > > > > > > > > > > > > > > > > - NoSQL persistence with InMemory
> > > backend.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Both alternatives test real
> > > > production-grade
> > > > > > > > > persistence
> > > > > > > > > > > code.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > And finally, TreeMapMetaStore is
> > > currently
> > > > > the
> > > > > > > > > default
> > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > > > persistence in
> application.properties;
> > > and
> > > > > the
> > > > > > > Helm
> > > > > > > > > > > chart also
> > > > > > > > > > > > > > > > > > > > advertises it as the default. These
> are
> > > not
> > > > > > sane
> > > > > > > > > > > defaults,
> > > > > > > > > > > > > > imho.
> > > > > > > > > > > > > > > It's
> > > > > > > > > > > > > > > > > > > > always tricky to provide a good
> default
> > > for
> > > > > > > > > datastores,
> > > > > > > > > > > but
> > > > > > > > > > > > > > > since JDBC
> > > > > > > > > > > > > > > > > > > > persistence is included by default in
> > the
> > > > > > server
> > > > > > > > > image,
> > > > > > > > > > > I think
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > including the H2 driver by default
> > could
> > > > give
> > > > > > us
> > > > > > > a
> > > > > > > > > saner
> > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > keeping the out-of-the-box experience
> > > > intact
> > > > > > (the
> > > > > > > > > > > license is
> > > > > > > > > > > > > > > Category
> > > > > > > > > > > > > > > > > > > > A).
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Concretely:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On the MetaStoreManagerFactory
> > hierarchy,
> > > > the
> > > > > > > > > following
> > > > > > > > > > > > > > > > > > > > implementations are completely
> > in-memory:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > -
> > InMemoryPolarisMetaStoreManagerFactory:
> > > > > could
> > > > > > > be
> > > > > > > > > > > removed
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > -
> > > > > > InMemoryAtomicOperationMetaStoreManagerFactory:
> > > > > > > > > could
> > > > > > > > > > > be
> > > > > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > -
> LocalPolarisMetaStoreManagerFactory:
> > is
> > > > the
> > > > > > > base
> > > > > > > > > class
> > > > > > > > > > > of the
> > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > above; imho it can be removed, but
> > since
> > > > it's
> > > > > > an
> > > > > > > > > abstract
> > > > > > > > > > > > > > class,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > may have been extended outside
> Polaris.
> > > But
> > > > > > > neither
> > > > > > > > > JDBC
> > > > > > > > > > > nor
> > > > > > > > > > > > > > > NoSQL use
> > > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On the PolarisMetaStoreManager
> > hierarchy:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > - TransactionalMetaStoreManagerImpl:
> is
> > > > only
> > > > > > > used by
> > > > > > > > > > > > > > > > > > > > LocalPolarisMetaStoreManagerFactory.
> > JDBC
> > > > and
> > > > > > > NoSQL
> > > > > > > > > do
> > > > > > > > > > > not use
> > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > Could be removed.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > -
> TransactionWorkspaceMetaStoreManager
> > > > > however
> > > > > > > is a
> > > > > > > > > > > different
> > > > > > > > > > > > > > > beast,
> > > > > > > > > > > > > > > > > > > > in spite of the similar name. It is
> in
> > > use
> > > > > > today
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > > commit
> > > > > > > > > > > > > > > path,
> > > > > > > > > > > > > > > > > > > > so should not be removed.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On the BasePersistence hierarchy:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > - TreeMapTransactionalPersistenceImpl
> > and
> > > > its
> > > > > > > > > > > TreeMapMetaStore
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > only used by
> > > > > > > InMemoryPolarisMetaStoreManagerFactory,
> > > > > > > > > and
> > > > > > > > > > > could
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > removed;
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > - TransactionalPersistence and
> > > > > > > > > > > > > > AbstractTransactionalPersistence:
> > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > are supertypes of
> > > > > > > TreeMapTransactionalPersistenceImpl
> > > > > > > > > > > and thus
> > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > > used for in-memory. They imo can be
> > > > removed,
> > > > > > but
> > > > > > > they
> > > > > > > > > > > might
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > extended or implemented outside
> > Polaris.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think it's important to keep
> Polaris
> > > code
> > > > > > tidy
> > > > > > > by
> > > > > > > > > > > removing
> > > > > > > > > > > > > > > unused,
> > > > > > > > > > > > > > > > > > > > unimplementable, or test-grade only
> > > > > components.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > What are your thoughts?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Posted by Yufei Gu <fl...@gmail.com>.
Dmitri, that's a valid concern, but not for the H2 lib only. Every library
introduced but not-used in production(e.g., test-only) could have CVEs. Do
we have data showing that the H2 lib tends to have more CVEs than others?
Yufei
On Wed, Jul 22, 2026 at 9:30 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi Yufei,
>
> The specific concerns were raised by Romain and Russell, as far as I can
> tell.
>
> Repeating the links to those messages for convenience: [1], [2].
>
> My personal interpretation is that it's not so much about CVEs that exist
> today, but about something that may be discovered in the future and impact
> the previously released Poalris images with the affected H2 driver
> versions.
>
> From my POV, it's a valid concern given that H2 is not actually meant for
> production use.
>
> WDYT?
>
> [1]: https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
>
> [2]: https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
>
> Cheers,
> Dmitri.
>
> On Tue, Jul 21, 2026 at 6:54 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > Could you help me understand the security concern? Every library can have
> > CVEs. Do you mean that the H2 library currently has unresolved CVEs, or
> > that it has historically had more CVEs than other libraries?
> >
> > Yufei
> >
> >
> > On Fri, Jul 17, 2026 at 11:18 AM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Yufei,
> > >
> > > What is your take on the security concerns associated with including
> the
> > H2
> > > JDBC driver in the standard Polaris images (highlighted by Alex in this
> > > thread on Jul 3) [1]?
> > >
> > > [1] https://lists.apache.org/thread/8v42g3rvmt74b9s6ccgy7y8kky2hfbjo
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Fri, Jul 17, 2026 at 2:06 PM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > > Hi Alex,
> > > >
> > > > Thanks for summarizing the discussion and separating the two
> > questions. I
> > > > think that helps move us toward consensus.
> > > >
> > > > First of all, I agree with the direction, but I'm not sure it's
> urgent.
> > > > TreeMapMetaStore isn't perfect, but we already have both JDBC + H2
> and
> > > > NoSQL + in-memory in place. That should already satisfy most of test
> > > > coverage.
> > > >
> > > > For question 1, I'd prefer Option 1a (JDBC + H2).
> > > >
> > > > My main reason is that it exercises the same persistence layer,
> > > transaction
> > > > handling, schema management, and configuration that most production
> > > > deployments use. That gives us a better out of the box experience for
> > > both
> > > > users and developers, while also increasing confidence that what
> works
> > > > locally will behave similarly in production.
> > > >
> > > > Do we have to wait for either PR 4812 or 4984? I think bundling the
> H2
> > > > driver should just unblock 1a.
> > > >
> > > > For question 2, I agree with handling test migration on a case by
> case
> > > > basis. Different tests have different goals, and using the most
> > > appropriate
> > > > backend for each test seems reasonable.
> > > >
> > > > Thanks,
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Fri, Jul 17, 2026 at 10:45 AM Dmitri Bourlatchkov <
> dimas@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Hi Alex,
> > > > >
> > > > > Option 1b certainly makes sense to me. I support going in that
> > > direction.
> > > > >
> > > > > I'd propose opening a PR for that merging it before proceeding to
> > test
> > > > code
> > > > > changes.
> > > > >
> > > > > In general, if 1b is acceptable, I'd think H2 should be removed [1]
> > > from
> > > > > tests and replaced with PostgreSQL or NoSQL InMemory. With that in
> > > mind,
> > > > > updating tests on a case-by-case basis seems reasonable to me. We
> can
> > > > > decide separately in each PR.
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > >
> > > > > On Fri, Jul 17, 2026 at 12:59 PM Alexandre Dutra <
> adutra@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Dmitri, hi all,
> > > > > >
> > > > > > Unfortunately I'm not sure your interpretation matches mine, so
> > let's
> > > > > > try to wrap up this discussion methodically.
> > > > > >
> > > > > > We agreed to deprecate TreeMapMetaStore for removal. But we
> haven't
> > > > > > reached consensus on two questions:
> > > > > >
> > > > > > 1) What should be the default metastore in the server image?
> > > > > > 2) How to replace TreeMapMetaStore in tests?
> > > > > >
> > > > > > Question 1) is tricky and needs a clear decision. We have only
> two
> > > > > > options for a smooth UX (aka, "docker run apache/polaris" just
> > > works):
> > > > > >
> > > > > > - Option 1a: JDBC+H2
> > > > > > - Option 1b: NoSQL+InMemory
> > > > > >
> > > > > > Option 1a is hard because it requires either my work on [4812]
> > > > > > (runtime activated datasources) or Yufei's work on [4984]
> > (unmanaged
> > > > > > datasources). It also *requires* the H2 driver to be bundled by
> > > > > > default, which some people are not comfortable with. In short, I
> > > don't
> > > > > > see this option being feasible any time soon.
> > > > > >
> > > > > > Therefore, I think option 1b is much easier, and that would be my
> > > > > > preference.
> > > > > >
> > > > > > Question 2) is not as urgent, as I'm fairly sure we can find a
> > > > > > solution as we go. I can see a few options:
> > > > > > - Option 2a: migrate to JDBC+H2
> > > > > > - Option 2b: migrate to NoSQL+InMemory
> > > > > > - Option 2c: decide on a case-by-case basis
> > > > > >
> > > > > > My personal preference would be 2c.
> > > > > >
> > > > > > Can we try to reach consensus on the two questions please?
> > > > > >
> > > > > > Thanks,
> > > > > > Alex
> > > > > >
> > > > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > > > > [4984]: https://github.com/apache/polaris/pull/4984
> > > > > >
> > > > > > On Mon, Jul 6, 2026 at 8:29 PM Dmitri Bourlatchkov <
> > dimas@apache.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > My interpretation of the discussion so far is that:
> > > > > > >
> > > > > > > 1) We do want to deprecate the TreeMap MetaStore for removal
> > > > > > >
> > > > > > > 2) JDBC + PostgreSQL should be used for testing as well as
> NoSQL
> > +
> > > > > > MongoDB
> > > > > > > (I believe it is in fact the case in current CI).
> > > > > > >
> > > > > > > 3) Instead of the TreeMap MetaStore, we will use the in-memory
> > > NoSQL
> > > > > > > MetaStore (for lightweight tests)
> > > > > > >
> > > > > > > Now, H2 becomes irrevant, IMHO, but removing it is being
> > discussed
> > > > > > > separately [1].
> > > > > > >
> > > > > > > How does that sound?
> > > > > > >
> > > > > > > [1]
> > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Fri, Jul 3, 2026 at 5:25 AM Alexandre Dutra <
> > adutra@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'm confused: security concerns were previously raised here
> [1]
> > > and
> > > > > > > > here [2] about bundling the H2 driver due to its past record
> of
> > > > > > > > critical CVEs.
> > > > > > > >
> > > > > > > > As a result, I assumed the JDBC+H2 option had been abandoned.
> > Has
> > > > > this
> > > > > > > > proposal been brought back into consideration?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Alex
> > > > > > > >
> > > > > > > > [1]:
> > > > > https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
> > > > > > > > [2]:
> > > > > https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 3, 2026 at 8:54 AM Jean-Baptiste Onofré <
> > > > jb@nanthrax.net
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi
> > > > > > > > >
> > > > > > > > > Yeah fully agree and it was my point when I replied on the
> > > thread
> > > > > > > > > initially.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > JB
> > > > > > > > >
> > > > > > > > > Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <
> flyrain000@gmail.com
> > >
> > > a
> > > > > > écrit :
> > > > > > > > >
> > > > > > > > > > If our goal is to make the default experience closer to
> > > > > production
> > > > > > > > > > while removing TreeMap, then JDBC with H2 seems like a
> more
> > > > > natural
> > > > > > > > > > choice. It exercises the same persistence layer, schema
> > > > > management,
> > > > > > > > > > and configuration that users will encounter in
> production,
> > > > while
> > > > > > still
> > > > > > > > > > keeping setup lightweight.
> > > > > > > > > >
> > > > > > > > > > Yufei
> > > > > > > > > >
> > > > > > > > > > Yufei
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <
> > > > > adutra@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Russell,
> > > > > > > > > > >
> > > > > > > > > > > The existing default metastore shouldn't be viewed
> merely
> > > as
> > > > a
> > > > > > > > > > > "relatively simple, self-contained implementation": it
> > has
> > > > > unique
> > > > > > > > > > > quirks, such as returning result types that no other
> > > > metastore
> > > > > > does.
> > > > > > > > > > >
> > > > > > > > > > > Choosing the status quo simply to avoid making changes
> is
> > > > imho
> > > > > > the
> > > > > > > > > > > worst solution.
> > > > > > > > > > >
> > > > > > > > > > > Transitioning to the NoSQL metastore as the default
> > > remains a
> > > > > net
> > > > > > > > > > > benefit, even if it is only used by 30% of the user
> base.
> > > > > > > > > > >
> > > > > > > > > > > However, I agree with keeping the test suite on JDBC +
> H2
> > > by
> > > > > > default
> > > > > > > > > > > if that ensures a higher level of confidence in our
> tests
> > > > > > ability to
> > > > > > > > > > > reproduce real production setups.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Alex
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
> > > > > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > I agree TreeMap and its associated transactional
> stack
> > > > should
> > > > > > be
> > > > > > > > > > deprecated.
> > > > > > > > > > > > It’s test-only, confusing, and not used by JDBC
> > > production
> > > > > > paths.
> > > > > > > > > > > >
> > > > > > > > > > > > I’m less convinced that NoSQL + InMemory is a clearly
> > > > > superior
> > > > > > > > > > replacement
> > > > > > > > > > > > for TreeMap in tests. At the storage layer it’s
> > > essentially
> > > > > the
> > > > > > > > same
> > > > > > > > > > class
> > > > > > > > > > > > of
> > > > > > > > > > > > thing: an in-memory, non-durable backend. The
> > difference
> > > > is
> > > > > > that
> > > > > > > > it
> > > > > > > > > > > > exercises
> > > > > > > > > > > > a completely different MetaStoreManager/persistence
> > stack
> > > > > > > > > > > > (NoSqlMetaStoreManager
> > > > > > > > > > > > vs TransactionalMetaStoreManagerImpl), not the JDBC
> > path
> > > > most
> > > > > > > > > > deployments
> > > > > > > > > > > > use. It
> > > > > > > > > > > > replaces a relatively simple, self-contained
> > > implementation
> > > > > > with
> > > > > > > > > > something
> > > > > > > > > > > > that is
> > > > > > > > > > > > part of a much more complicated persistence stack,
> > while
> > > > > > remaining
> > > > > > > > > > > > non-durable
> > > > > > > > > > > > and test-only at the storage layer.
> > > > > > > > > > > >
> > > > > > > > > > > > For the problems originally raised (test-grade
> > defaults,
> > > > dead
> > > > > > > > > > production
> > > > > > > > > > > > path, developer confusion),
> > > > > > > > > > > > JDBC + H2 seems like an actual fix.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <
> > > > > > > > dimas@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I tend to agree that the DataSource discussion
> > related
> > > to
> > > > > H2
> > > > > > > > became
> > > > > > > > > > a bit
> > > > > > > > > > > > > convoluted, and it's a lot simpler to use the NoSQL
> > > > > in-memory
> > > > > > > > > > persistence
> > > > > > > > > > > > > instead.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <
> > > > > > > > adutra@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Some updates on this:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I recently started a separate discussion
> regarding
> > > > > dynamic
> > > > > > > > > > datasource
> > > > > > > > > > > > > > activation at runtime [1]. This was intended as a
> > > > > > prerequisite
> > > > > > > > for
> > > > > > > > > > > > > > adopting H2 as the default JDBC driver, as
> > suggested
> > > on
> > > > > > this
> > > > > > > > > > thread.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > However, that proposal may end up being rejected.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Given this situation, we should reconsider our
> > > options.
> > > > > If
> > > > > > the
> > > > > > > > > > > > > > PostgreSQL + H2 approach is no longer a viable
> > > > > replacement
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > > > TreeMapMetaStore, I propose an alternative: let's
> > use
> > > > the
> > > > > > NoSQL
> > > > > > > > > > > > > > metastore with an InMemory backend.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > My reasoning is as follows:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. It remains a production-ready metastore, so
> > > strictly
> > > > > > > > superior to
> > > > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. The metastore state update logic is the same
> > > > > regardless
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > > > backend, so we're still close to a real
> production
> > > > setup
> > > > > > (same
> > > > > > > > as
> > > > > > > > > > with
> > > > > > > > > > > > > > an in-memory JDBC driver).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. It doesn't require any external dependencies
> (no
> > > > > bundled
> > > > > > > > > > driver).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'd also note that while the NoSQL metastore is
> > > already
> > > > > > > > included
> > > > > > > > > > > > > > today, it is not actionable because no backend is
> > > > > > available.
> > > > > > > > This
> > > > > > > > > > goes
> > > > > > > > > > > > > > against the onboarding UX that we're trying to
> > > achieve,
> > > > > as
> > > > > > > > users
> > > > > > > > > > > > > > interested in Polaris with the NoSQL metastore
> > cannot
> > > > > test
> > > > > > this
> > > > > > > > > > setup
> > > > > > > > > > > > > > with the official image.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Therefore, I think NoSQL + InMemory would serve
> as
> > a
> > > > more
> > > > > > > > > > appropriate
> > > > > > > > > > > > > > default setup for the official Polaris image.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > From what I see, it would boil down to adding one
> > > line
> > > > to
> > > > > > > > > > > > > > polaris-server or polaris-service:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > What are your thoughts on this?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]:
> > > > > > > > > >
> > > > https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra
> <
> > > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I wanted to give an update on this topic: I
> have
> > > > > started
> > > > > > > > working
> > > > > > > > > > on
> > > > > > > > > > > > > > > this, but I would like to have [4812] merged
> > first,
> > > > > since
> > > > > > > > that's
> > > > > > > > > > a
> > > > > > > > > > > > > > > prerequisite.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > [4812]:
> > > https://github.com/apache/polaris/pull/4812
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> > > > > > > > > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I believe the original idea of
> TreeMapMetastore
> > > was
> > > > > > that it
> > > > > > > > > > looked
> > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > similar to FoundationDB from an API
> > perspective,
> > > > so
> > > > > it
> > > > > > > > served
> > > > > > > > > > well
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > test system for the original backend being
> > > > developed
> > > > > > at SF.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I agree that H2 + JDBC makes sense to me now
> > for
> > > > the
> > > > > > > > project.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre
> > Dutra
> > > <
> > > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thank you for your feedback.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There seems to be a general agreement on
> the
> > > idea
> > > > > of
> > > > > > > > > > deprecating
> > > > > > > > > > > > > > > > > TreeMapMetaStore, coupled with the JDBC +
> H2
> > > > > > solution for
> > > > > > > > > > tests.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > In most modules, the tests migration won't
> > pose
> > > > any
> > > > > > > > serious
> > > > > > > > > > > > > > > > > challenges, as replacing the metastore is
> > > > generally
> > > > > > just
> > > > > > > > a
> > > > > > > > > > matter
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > changing the configuration, and making sure
> > the
> > > > > > realm is
> > > > > > > > > > properly
> > > > > > > > > > > > > > > > > bootstrapped.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There will be, however, a few tricky
> > situations
> > > > in
> > > > > > > > > > polaris-core: a
> > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > tests rely on the TreeMapMetaStore, mostly
> > as a
> > > > > test
> > > > > > > > > > convenience;
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > there is no obvious replacement for it in
> > that
> > > > > > module. I
> > > > > > > > am
> > > > > > > > > > however
> > > > > > > > > > > > > > > > > confident that we can find a solution for
> > that,
> > > > > > either
> > > > > > > > based
> > > > > > > > > > on
> > > > > > > > > > > > > > mocks,
> > > > > > > > > > > > > > > > > or by bringing in a real metastore.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > If no objections are raised, I am going to
> > > > prepare
> > > > > a
> > > > > > PR
> > > > > > > > for
> > > > > > > > > > this.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri
> > > > > Bourlatchkov <
> > > > > > > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for starting this thread.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I also find the
> > > > TransactionalMetaStoreManagerImpl
> > > > > > and
> > > > > > > > > > related
> > > > > > > > > > > > > call
> > > > > > > > > > > > > > paths
> > > > > > > > > > > > > > > > > > conuising in Apache Polaris code.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I support promoting H2 to the default (in
> > > > memory)
> > > > > > > > > > Persistence
> > > > > > > > > > > > > > backend for
> > > > > > > > > > > > > > > > > > getting started cases. This should also
> > > resolve
> > > > > the
> > > > > > > > old H2
> > > > > > > > > > > > > > evolution
> > > > > > > > > > > > > > > > > thread
> > > > > > > > > > > > > > > > > > [1] by ensuring it is used regularly on
> the
> > > > same
> > > > > > code
> > > > > > > > > > paths as
> > > > > > > > > > > > > > > > > PostgreSQL.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Deprecating
> > TransactionalMetaStoreManagerImpl
> > > > for
> > > > > > > > removal
> > > > > > > > > > also
> > > > > > > > > > > > > > sounds
> > > > > > > > > > > > > > > > > > reasonable to me. Existing downstream
> users
> > > > > > > > > > > > > > > > > > of TransactionalMetaStoreManagerImpl will
> > > have
> > > > > > time to
> > > > > > > > > > migrate,
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > even
> > > > > > > > > > > > > > > > > > embed that code (per ASF license) into
> > local
> > > > > > builds,
> > > > > > > > > > during the
> > > > > > > > > > > > > > > > > deprecation
> > > > > > > > > > > > > > > > > > phase.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > >
> > > > > > > >
> > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM
> Alexandre
> > > > Dutra
> > > > > <
> > > > > > > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I am writing to ask the community
> whether
> > > it
> > > > is
> > > > > > OK to
> > > > > > > > > > deprecate
> > > > > > > > > > > > > > > > > > > TreeMapMetaStore, as well as all the
> > > > in-memory
> > > > > > > > metastore
> > > > > > > > > > > > > manager,
> > > > > > > > > > > > > > > > > > > metastore manager factory, and
> > persistence
> > > > > types
> > > > > > that
> > > > > > > > > > rely
> > > > > > > > > > > > > > solely on
> > > > > > > > > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > As we know, these components are
> > test-grade
> > > > > only,
> > > > > > > > and not
> > > > > > > > > > > > > > suitable for
> > > > > > > > > > > > > > > > > > > production. They trigger a production
> > > > readiness
> > > > > > > > alert on
> > > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > > > > startup. It's a considerable amount of
> > code
> > > > > that
> > > > > > is
> > > > > > > > > > virtually
> > > > > > > > > > > > > > dead in
> > > > > > > > > > > > > > > > > > > production.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It's also confusing for developers.
> E.g.
> > > the
> > > > > > > > > > "transactional"
> > > > > > > > > > > > > > metastore
> > > > > > > > > > > > > > > > > > > is not transactional in the JDBC sense
> of
> > > the
> > > > > > term,
> > > > > > > > and
> > > > > > > > > > thus
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > by JDBC persistence. It also has its
> > > quirks:
> > > > > some
> > > > > > > > return
> > > > > > > > > > > > > > statuses are
> > > > > > > > > > > > > > > > > > > only returned by that manager.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > However, these components are used in
> > > tests,
> > > > > and
> > > > > > I
> > > > > > > > agree
> > > > > > > > > > that
> > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > > > > > > useful to have an in-memory version of
> > the
> > > > > > > > persistence
> > > > > > > > > > layer
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > tests. But we have today two
> alternatives
> > > > that
> > > > > > are
> > > > > > > > imho
> > > > > > > > > > > > > superior
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > tests in polaris-runtime-service:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > - JDBC persistence with H2 backend.
> There
> > > are
> > > > > > > > already a
> > > > > > > > > > few
> > > > > > > > > > > > > tests
> > > > > > > > > > > > > > > > > > > using this setup.
> > > > > > > > > > > > > > > > > > > - NoSQL persistence with InMemory
> > backend.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Both alternatives test real
> > > production-grade
> > > > > > > > persistence
> > > > > > > > > > code.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > And finally, TreeMapMetaStore is
> > currently
> > > > the
> > > > > > > > default
> > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > > persistence in application.properties;
> > and
> > > > the
> > > > > > Helm
> > > > > > > > > > chart also
> > > > > > > > > > > > > > > > > > > advertises it as the default. These are
> > not
> > > > > sane
> > > > > > > > > > defaults,
> > > > > > > > > > > > > imho.
> > > > > > > > > > > > > > It's
> > > > > > > > > > > > > > > > > > > always tricky to provide a good default
> > for
> > > > > > > > datastores,
> > > > > > > > > > but
> > > > > > > > > > > > > > since JDBC
> > > > > > > > > > > > > > > > > > > persistence is included by default in
> the
> > > > > server
> > > > > > > > image,
> > > > > > > > > > I think
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > including the H2 driver by default
> could
> > > give
> > > > > us
> > > > > > a
> > > > > > > > saner
> > > > > > > > > > > > > default
> > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > keeping the out-of-the-box experience
> > > intact
> > > > > (the
> > > > > > > > > > license is
> > > > > > > > > > > > > > Category
> > > > > > > > > > > > > > > > > > > A).
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Concretely:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On the MetaStoreManagerFactory
> hierarchy,
> > > the
> > > > > > > > following
> > > > > > > > > > > > > > > > > > > implementations are completely
> in-memory:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > -
> InMemoryPolarisMetaStoreManagerFactory:
> > > > could
> > > > > > be
> > > > > > > > > > removed
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > -
> > > > > InMemoryAtomicOperationMetaStoreManagerFactory:
> > > > > > > > could
> > > > > > > > > > be
> > > > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > - LocalPolarisMetaStoreManagerFactory:
> is
> > > the
> > > > > > base
> > > > > > > > class
> > > > > > > > > > of the
> > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > above; imho it can be removed, but
> since
> > > it's
> > > > > an
> > > > > > > > abstract
> > > > > > > > > > > > > class,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > may have been extended outside Polaris.
> > But
> > > > > > neither
> > > > > > > > JDBC
> > > > > > > > > > nor
> > > > > > > > > > > > > > NoSQL use
> > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On the PolarisMetaStoreManager
> hierarchy:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > - TransactionalMetaStoreManagerImpl: is
> > > only
> > > > > > used by
> > > > > > > > > > > > > > > > > > > LocalPolarisMetaStoreManagerFactory.
> JDBC
> > > and
> > > > > > NoSQL
> > > > > > > > do
> > > > > > > > > > not use
> > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > Could be removed.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > - TransactionWorkspaceMetaStoreManager
> > > > however
> > > > > > is a
> > > > > > > > > > different
> > > > > > > > > > > > > > beast,
> > > > > > > > > > > > > > > > > > > in spite of the similar name. It is in
> > use
> > > > > today
> > > > > > on
> > > > > > > > the
> > > > > > > > > > commit
> > > > > > > > > > > > > > path,
> > > > > > > > > > > > > > > > > > > so should not be removed.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On the BasePersistence hierarchy:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > - TreeMapTransactionalPersistenceImpl
> and
> > > its
> > > > > > > > > > TreeMapMetaStore
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > only used by
> > > > > > InMemoryPolarisMetaStoreManagerFactory,
> > > > > > > > and
> > > > > > > > > > could
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > removed;
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > - TransactionalPersistence and
> > > > > > > > > > > > > AbstractTransactionalPersistence:
> > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > are supertypes of
> > > > > > TreeMapTransactionalPersistenceImpl
> > > > > > > > > > and thus
> > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > used for in-memory. They imo can be
> > > removed,
> > > > > but
> > > > > > they
> > > > > > > > > > might
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > extended or implemented outside
> Polaris.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think it's important to keep Polaris
> > code
> > > > > tidy
> > > > > > by
> > > > > > > > > > removing
> > > > > > > > > > > > > > unused,
> > > > > > > > > > > > > > > > > > > unimplementable, or test-grade only
> > > > components.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > What are your thoughts?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Yufei,
The specific concerns were raised by Romain and Russell, as far as I can
tell.
Repeating the links to those messages for convenience: [1], [2].
My personal interpretation is that it's not so much about CVEs that exist
today, but about something that may be discovered in the future and impact
the previously released Poalris images with the affected H2 driver versions.
From my POV, it's a valid concern given that H2 is not actually meant for
production use.
WDYT?
[1]: https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
[2]: https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
Cheers,
Dmitri.
On Tue, Jul 21, 2026 at 6:54 PM Yufei Gu <fl...@gmail.com> wrote:
> Could you help me understand the security concern? Every library can have
> CVEs. Do you mean that the H2 library currently has unresolved CVEs, or
> that it has historically had more CVEs than other libraries?
>
> Yufei
>
>
> On Fri, Jul 17, 2026 at 11:18 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Yufei,
> >
> > What is your take on the security concerns associated with including the
> H2
> > JDBC driver in the standard Polaris images (highlighted by Alex in this
> > thread on Jul 3) [1]?
> >
> > [1] https://lists.apache.org/thread/8v42g3rvmt74b9s6ccgy7y8kky2hfbjo
> >
> > Thanks,
> > Dmitri.
> >
> > On Fri, Jul 17, 2026 at 2:06 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Hi Alex,
> > >
> > > Thanks for summarizing the discussion and separating the two
> questions. I
> > > think that helps move us toward consensus.
> > >
> > > First of all, I agree with the direction, but I'm not sure it's urgent.
> > > TreeMapMetaStore isn't perfect, but we already have both JDBC + H2 and
> > > NoSQL + in-memory in place. That should already satisfy most of test
> > > coverage.
> > >
> > > For question 1, I'd prefer Option 1a (JDBC + H2).
> > >
> > > My main reason is that it exercises the same persistence layer,
> > transaction
> > > handling, schema management, and configuration that most production
> > > deployments use. That gives us a better out of the box experience for
> > both
> > > users and developers, while also increasing confidence that what works
> > > locally will behave similarly in production.
> > >
> > > Do we have to wait for either PR 4812 or 4984? I think bundling the H2
> > > driver should just unblock 1a.
> > >
> > > For question 2, I agree with handling test migration on a case by case
> > > basis. Different tests have different goals, and using the most
> > appropriate
> > > backend for each test seems reasonable.
> > >
> > > Thanks,
> > >
> > > Yufei
> > >
> > >
> > > On Fri, Jul 17, 2026 at 10:45 AM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > wrote:
> > >
> > > > Hi Alex,
> > > >
> > > > Option 1b certainly makes sense to me. I support going in that
> > direction.
> > > >
> > > > I'd propose opening a PR for that merging it before proceeding to
> test
> > > code
> > > > changes.
> > > >
> > > > In general, if 1b is acceptable, I'd think H2 should be removed [1]
> > from
> > > > tests and replaced with PostgreSQL or NoSQL InMemory. With that in
> > mind,
> > > > updating tests on a case-by-case basis seems reasonable to me. We can
> > > > decide separately in each PR.
> > > >
> > > > [1] https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > >
> > > > On Fri, Jul 17, 2026 at 12:59 PM Alexandre Dutra <ad...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Dmitri, hi all,
> > > > >
> > > > > Unfortunately I'm not sure your interpretation matches mine, so
> let's
> > > > > try to wrap up this discussion methodically.
> > > > >
> > > > > We agreed to deprecate TreeMapMetaStore for removal. But we haven't
> > > > > reached consensus on two questions:
> > > > >
> > > > > 1) What should be the default metastore in the server image?
> > > > > 2) How to replace TreeMapMetaStore in tests?
> > > > >
> > > > > Question 1) is tricky and needs a clear decision. We have only two
> > > > > options for a smooth UX (aka, "docker run apache/polaris" just
> > works):
> > > > >
> > > > > - Option 1a: JDBC+H2
> > > > > - Option 1b: NoSQL+InMemory
> > > > >
> > > > > Option 1a is hard because it requires either my work on [4812]
> > > > > (runtime activated datasources) or Yufei's work on [4984]
> (unmanaged
> > > > > datasources). It also *requires* the H2 driver to be bundled by
> > > > > default, which some people are not comfortable with. In short, I
> > don't
> > > > > see this option being feasible any time soon.
> > > > >
> > > > > Therefore, I think option 1b is much easier, and that would be my
> > > > > preference.
> > > > >
> > > > > Question 2) is not as urgent, as I'm fairly sure we can find a
> > > > > solution as we go. I can see a few options:
> > > > > - Option 2a: migrate to JDBC+H2
> > > > > - Option 2b: migrate to NoSQL+InMemory
> > > > > - Option 2c: decide on a case-by-case basis
> > > > >
> > > > > My personal preference would be 2c.
> > > > >
> > > > > Can we try to reach consensus on the two questions please?
> > > > >
> > > > > Thanks,
> > > > > Alex
> > > > >
> > > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > > > [4984]: https://github.com/apache/polaris/pull/4984
> > > > >
> > > > > On Mon, Jul 6, 2026 at 8:29 PM Dmitri Bourlatchkov <
> dimas@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > My interpretation of the discussion so far is that:
> > > > > >
> > > > > > 1) We do want to deprecate the TreeMap MetaStore for removal
> > > > > >
> > > > > > 2) JDBC + PostgreSQL should be used for testing as well as NoSQL
> +
> > > > > MongoDB
> > > > > > (I believe it is in fact the case in current CI).
> > > > > >
> > > > > > 3) Instead of the TreeMap MetaStore, we will use the in-memory
> > NoSQL
> > > > > > MetaStore (for lightweight tests)
> > > > > >
> > > > > > Now, H2 becomes irrevant, IMHO, but removing it is being
> discussed
> > > > > > separately [1].
> > > > > >
> > > > > > How does that sound?
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Fri, Jul 3, 2026 at 5:25 AM Alexandre Dutra <
> adutra@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'm confused: security concerns were previously raised here [1]
> > and
> > > > > > > here [2] about bundling the H2 driver due to its past record of
> > > > > > > critical CVEs.
> > > > > > >
> > > > > > > As a result, I assumed the JDBC+H2 option had been abandoned.
> Has
> > > > this
> > > > > > > proposal been brought back into consideration?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Alex
> > > > > > >
> > > > > > > [1]:
> > > > https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
> > > > > > > [2]:
> > > > https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 3, 2026 at 8:54 AM Jean-Baptiste Onofré <
> > > jb@nanthrax.net
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi
> > > > > > > >
> > > > > > > > Yeah fully agree and it was my point when I replied on the
> > thread
> > > > > > > > initially.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > JB
> > > > > > > >
> > > > > > > > Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <flyrain000@gmail.com
> >
> > a
> > > > > écrit :
> > > > > > > >
> > > > > > > > > If our goal is to make the default experience closer to
> > > > production
> > > > > > > > > while removing TreeMap, then JDBC with H2 seems like a more
> > > > natural
> > > > > > > > > choice. It exercises the same persistence layer, schema
> > > > management,
> > > > > > > > > and configuration that users will encounter in production,
> > > while
> > > > > still
> > > > > > > > > keeping setup lightweight.
> > > > > > > > >
> > > > > > > > > Yufei
> > > > > > > > >
> > > > > > > > > Yufei
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <
> > > > adutra@apache.org>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Russell,
> > > > > > > > > >
> > > > > > > > > > The existing default metastore shouldn't be viewed merely
> > as
> > > a
> > > > > > > > > > "relatively simple, self-contained implementation": it
> has
> > > > unique
> > > > > > > > > > quirks, such as returning result types that no other
> > > metastore
> > > > > does.
> > > > > > > > > >
> > > > > > > > > > Choosing the status quo simply to avoid making changes is
> > > imho
> > > > > the
> > > > > > > > > > worst solution.
> > > > > > > > > >
> > > > > > > > > > Transitioning to the NoSQL metastore as the default
> > remains a
> > > > net
> > > > > > > > > > benefit, even if it is only used by 30% of the user base.
> > > > > > > > > >
> > > > > > > > > > However, I agree with keeping the test suite on JDBC + H2
> > by
> > > > > default
> > > > > > > > > > if that ensures a higher level of confidence in our tests
> > > > > ability to
> > > > > > > > > > reproduce real production setups.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Alex
> > > > > > > > > >
> > > > > > > > > > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
> > > > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > I agree TreeMap and its associated transactional stack
> > > should
> > > > > be
> > > > > > > > > deprecated.
> > > > > > > > > > > It’s test-only, confusing, and not used by JDBC
> > production
> > > > > paths.
> > > > > > > > > > >
> > > > > > > > > > > I’m less convinced that NoSQL + InMemory is a clearly
> > > > superior
> > > > > > > > > replacement
> > > > > > > > > > > for TreeMap in tests. At the storage layer it’s
> > essentially
> > > > the
> > > > > > > same
> > > > > > > > > class
> > > > > > > > > > > of
> > > > > > > > > > > thing: an in-memory, non-durable backend. The
> difference
> > > is
> > > > > that
> > > > > > > it
> > > > > > > > > > > exercises
> > > > > > > > > > > a completely different MetaStoreManager/persistence
> stack
> > > > > > > > > > > (NoSqlMetaStoreManager
> > > > > > > > > > > vs TransactionalMetaStoreManagerImpl), not the JDBC
> path
> > > most
> > > > > > > > > deployments
> > > > > > > > > > > use. It
> > > > > > > > > > > replaces a relatively simple, self-contained
> > implementation
> > > > > with
> > > > > > > > > something
> > > > > > > > > > > that is
> > > > > > > > > > > part of a much more complicated persistence stack,
> while
> > > > > remaining
> > > > > > > > > > > non-durable
> > > > > > > > > > > and test-only at the storage layer.
> > > > > > > > > > >
> > > > > > > > > > > For the problems originally raised (test-grade
> defaults,
> > > dead
> > > > > > > > > production
> > > > > > > > > > > path, developer confusion),
> > > > > > > > > > > JDBC + H2 seems like an actual fix.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <
> > > > > > > dimas@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > >
> > > > > > > > > > > > I tend to agree that the DataSource discussion
> related
> > to
> > > > H2
> > > > > > > became
> > > > > > > > > a bit
> > > > > > > > > > > > convoluted, and it's a lot simpler to use the NoSQL
> > > > in-memory
> > > > > > > > > persistence
> > > > > > > > > > > > instead.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Dmitri.
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <
> > > > > > > adutra@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Some updates on this:
> > > > > > > > > > > > >
> > > > > > > > > > > > > I recently started a separate discussion regarding
> > > > dynamic
> > > > > > > > > datasource
> > > > > > > > > > > > > activation at runtime [1]. This was intended as a
> > > > > prerequisite
> > > > > > > for
> > > > > > > > > > > > > adopting H2 as the default JDBC driver, as
> suggested
> > on
> > > > > this
> > > > > > > > > thread.
> > > > > > > > > > > > >
> > > > > > > > > > > > > However, that proposal may end up being rejected.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Given this situation, we should reconsider our
> > options.
> > > > If
> > > > > the
> > > > > > > > > > > > > PostgreSQL + H2 approach is no longer a viable
> > > > replacement
> > > > > to
> > > > > > > the
> > > > > > > > > > > > > TreeMapMetaStore, I propose an alternative: let's
> use
> > > the
> > > > > NoSQL
> > > > > > > > > > > > > metastore with an InMemory backend.
> > > > > > > > > > > > >
> > > > > > > > > > > > > My reasoning is as follows:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. It remains a production-ready metastore, so
> > strictly
> > > > > > > superior to
> > > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. The metastore state update logic is the same
> > > > regardless
> > > > > of
> > > > > > > the
> > > > > > > > > > > > > backend, so we're still close to a real production
> > > setup
> > > > > (same
> > > > > > > as
> > > > > > > > > with
> > > > > > > > > > > > > an in-memory JDBC driver).
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3. It doesn't require any external dependencies (no
> > > > bundled
> > > > > > > > > driver).
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'd also note that while the NoSQL metastore is
> > already
> > > > > > > included
> > > > > > > > > > > > > today, it is not actionable because no backend is
> > > > > available.
> > > > > > > This
> > > > > > > > > goes
> > > > > > > > > > > > > against the onboarding UX that we're trying to
> > achieve,
> > > > as
> > > > > > > users
> > > > > > > > > > > > > interested in Polaris with the NoSQL metastore
> cannot
> > > > test
> > > > > this
> > > > > > > > > setup
> > > > > > > > > > > > > with the official image.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Therefore, I think NoSQL + InMemory would serve as
> a
> > > more
> > > > > > > > > appropriate
> > > > > > > > > > > > > default setup for the official Polaris image.
> > > > > > > > > > > > >
> > > > > > > > > > > > > From what I see, it would boil down to adding one
> > line
> > > to
> > > > > > > > > > > > > polaris-server or polaris-service:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > > > > > > > > > > > >
> > > > > > > > > > > > > What are your thoughts on this?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Alex
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]:
> > > > > > > > >
> > > https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra <
> > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I wanted to give an update on this topic: I have
> > > > started
> > > > > > > working
> > > > > > > > > on
> > > > > > > > > > > > > > this, but I would like to have [4812] merged
> first,
> > > > since
> > > > > > > that's
> > > > > > > > > a
> > > > > > > > > > > > > > prerequisite.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [4812]:
> > https://github.com/apache/polaris/pull/4812
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> > > > > > > > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I believe the original idea of TreeMapMetastore
> > was
> > > > > that it
> > > > > > > > > looked
> > > > > > > > > > > > very
> > > > > > > > > > > > > > > similar to FoundationDB from an API
> perspective,
> > > so
> > > > it
> > > > > > > served
> > > > > > > > > well
> > > > > > > > > > > > as
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > test system for the original backend being
> > > developed
> > > > > at SF.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I agree that H2 + JDBC makes sense to me now
> for
> > > the
> > > > > > > project.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre
> Dutra
> > <
> > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you for your feedback.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There seems to be a general agreement on the
> > idea
> > > > of
> > > > > > > > > deprecating
> > > > > > > > > > > > > > > > TreeMapMetaStore, coupled with the JDBC + H2
> > > > > solution for
> > > > > > > > > tests.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > In most modules, the tests migration won't
> pose
> > > any
> > > > > > > serious
> > > > > > > > > > > > > > > > challenges, as replacing the metastore is
> > > generally
> > > > > just
> > > > > > > a
> > > > > > > > > matter
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > changing the configuration, and making sure
> the
> > > > > realm is
> > > > > > > > > properly
> > > > > > > > > > > > > > > > bootstrapped.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There will be, however, a few tricky
> situations
> > > in
> > > > > > > > > polaris-core: a
> > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > tests rely on the TreeMapMetaStore, mostly
> as a
> > > > test
> > > > > > > > > convenience;
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > > there is no obvious replacement for it in
> that
> > > > > module. I
> > > > > > > am
> > > > > > > > > however
> > > > > > > > > > > > > > > > confident that we can find a solution for
> that,
> > > > > either
> > > > > > > based
> > > > > > > > > on
> > > > > > > > > > > > > mocks,
> > > > > > > > > > > > > > > > or by bringing in a real metastore.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > If no objections are raised, I am going to
> > > prepare
> > > > a
> > > > > PR
> > > > > > > for
> > > > > > > > > this.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri
> > > > Bourlatchkov <
> > > > > > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for starting this thread.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I also find the
> > > TransactionalMetaStoreManagerImpl
> > > > > and
> > > > > > > > > related
> > > > > > > > > > > > call
> > > > > > > > > > > > > paths
> > > > > > > > > > > > > > > > > conuising in Apache Polaris code.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I support promoting H2 to the default (in
> > > memory)
> > > > > > > > > Persistence
> > > > > > > > > > > > > backend for
> > > > > > > > > > > > > > > > > getting started cases. This should also
> > resolve
> > > > the
> > > > > > > old H2
> > > > > > > > > > > > > evolution
> > > > > > > > > > > > > > > > thread
> > > > > > > > > > > > > > > > > [1] by ensuring it is used regularly on the
> > > same
> > > > > code
> > > > > > > > > paths as
> > > > > > > > > > > > > > > > PostgreSQL.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Deprecating
> TransactionalMetaStoreManagerImpl
> > > for
> > > > > > > removal
> > > > > > > > > also
> > > > > > > > > > > > > sounds
> > > > > > > > > > > > > > > > > reasonable to me. Existing downstream users
> > > > > > > > > > > > > > > > > of TransactionalMetaStoreManagerImpl will
> > have
> > > > > time to
> > > > > > > > > migrate,
> > > > > > > > > > > > or
> > > > > > > > > > > > > even
> > > > > > > > > > > > > > > > > embed that code (per ASF license) into
> local
> > > > > builds,
> > > > > > > > > during the
> > > > > > > > > > > > > > > > deprecation
> > > > > > > > > > > > > > > > > phase.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > >
> > > > > > >
> https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre
> > > Dutra
> > > > <
> > > > > > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I am writing to ask the community whether
> > it
> > > is
> > > > > OK to
> > > > > > > > > deprecate
> > > > > > > > > > > > > > > > > > TreeMapMetaStore, as well as all the
> > > in-memory
> > > > > > > metastore
> > > > > > > > > > > > manager,
> > > > > > > > > > > > > > > > > > metastore manager factory, and
> persistence
> > > > types
> > > > > that
> > > > > > > > > rely
> > > > > > > > > > > > > solely on
> > > > > > > > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > As we know, these components are
> test-grade
> > > > only,
> > > > > > > and not
> > > > > > > > > > > > > suitable for
> > > > > > > > > > > > > > > > > > production. They trigger a production
> > > readiness
> > > > > > > alert on
> > > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > > > startup. It's a considerable amount of
> code
> > > > that
> > > > > is
> > > > > > > > > virtually
> > > > > > > > > > > > > dead in
> > > > > > > > > > > > > > > > > > production.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It's also confusing for developers. E.g.
> > the
> > > > > > > > > "transactional"
> > > > > > > > > > > > > metastore
> > > > > > > > > > > > > > > > > > is not transactional in the JDBC sense of
> > the
> > > > > term,
> > > > > > > and
> > > > > > > > > thus
> > > > > > > > > > > > not
> > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > by JDBC persistence. It also has its
> > quirks:
> > > > some
> > > > > > > return
> > > > > > > > > > > > > statuses are
> > > > > > > > > > > > > > > > > > only returned by that manager.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > However, these components are used in
> > tests,
> > > > and
> > > > > I
> > > > > > > agree
> > > > > > > > > that
> > > > > > > > > > > > > it's
> > > > > > > > > > > > > > > > > > useful to have an in-memory version of
> the
> > > > > > > persistence
> > > > > > > > > layer
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > tests. But we have today two alternatives
> > > that
> > > > > are
> > > > > > > imho
> > > > > > > > > > > > superior
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > tests in polaris-runtime-service:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > - JDBC persistence with H2 backend. There
> > are
> > > > > > > already a
> > > > > > > > > few
> > > > > > > > > > > > tests
> > > > > > > > > > > > > > > > > > using this setup.
> > > > > > > > > > > > > > > > > > - NoSQL persistence with InMemory
> backend.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Both alternatives test real
> > production-grade
> > > > > > > persistence
> > > > > > > > > code.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > And finally, TreeMapMetaStore is
> currently
> > > the
> > > > > > > default
> > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > persistence in application.properties;
> and
> > > the
> > > > > Helm
> > > > > > > > > chart also
> > > > > > > > > > > > > > > > > > advertises it as the default. These are
> not
> > > > sane
> > > > > > > > > defaults,
> > > > > > > > > > > > imho.
> > > > > > > > > > > > > It's
> > > > > > > > > > > > > > > > > > always tricky to provide a good default
> for
> > > > > > > datastores,
> > > > > > > > > but
> > > > > > > > > > > > > since JDBC
> > > > > > > > > > > > > > > > > > persistence is included by default in the
> > > > server
> > > > > > > image,
> > > > > > > > > I think
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > including the H2 driver by default could
> > give
> > > > us
> > > > > a
> > > > > > > saner
> > > > > > > > > > > > default
> > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > keeping the out-of-the-box experience
> > intact
> > > > (the
> > > > > > > > > license is
> > > > > > > > > > > > > Category
> > > > > > > > > > > > > > > > > > A).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Concretely:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On the MetaStoreManagerFactory hierarchy,
> > the
> > > > > > > following
> > > > > > > > > > > > > > > > > > implementations are completely in-memory:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > - InMemoryPolarisMetaStoreManagerFactory:
> > > could
> > > > > be
> > > > > > > > > removed
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > -
> > > > InMemoryAtomicOperationMetaStoreManagerFactory:
> > > > > > > could
> > > > > > > > > be
> > > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > - LocalPolarisMetaStoreManagerFactory: is
> > the
> > > > > base
> > > > > > > class
> > > > > > > > > of the
> > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > above; imho it can be removed, but since
> > it's
> > > > an
> > > > > > > abstract
> > > > > > > > > > > > class,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > may have been extended outside Polaris.
> But
> > > > > neither
> > > > > > > JDBC
> > > > > > > > > nor
> > > > > > > > > > > > > NoSQL use
> > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On the PolarisMetaStoreManager hierarchy:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > - TransactionalMetaStoreManagerImpl: is
> > only
> > > > > used by
> > > > > > > > > > > > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC
> > and
> > > > > NoSQL
> > > > > > > do
> > > > > > > > > not use
> > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > Could be removed.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > - TransactionWorkspaceMetaStoreManager
> > > however
> > > > > is a
> > > > > > > > > different
> > > > > > > > > > > > > beast,
> > > > > > > > > > > > > > > > > > in spite of the similar name. It is in
> use
> > > > today
> > > > > on
> > > > > > > the
> > > > > > > > > commit
> > > > > > > > > > > > > path,
> > > > > > > > > > > > > > > > > > so should not be removed.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On the BasePersistence hierarchy:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > - TreeMapTransactionalPersistenceImpl and
> > its
> > > > > > > > > TreeMapMetaStore
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > only used by
> > > > > InMemoryPolarisMetaStoreManagerFactory,
> > > > > > > and
> > > > > > > > > could
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > removed;
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > - TransactionalPersistence and
> > > > > > > > > > > > AbstractTransactionalPersistence:
> > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > are supertypes of
> > > > > TreeMapTransactionalPersistenceImpl
> > > > > > > > > and thus
> > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > used for in-memory. They imo can be
> > removed,
> > > > but
> > > > > they
> > > > > > > > > might
> > > > > > > > > > > > have
> > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > extended or implemented outside Polaris.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think it's important to keep Polaris
> code
> > > > tidy
> > > > > by
> > > > > > > > > removing
> > > > > > > > > > > > > unused,
> > > > > > > > > > > > > > > > > > unimplementable, or test-grade only
> > > components.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > What are your thoughts?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Posted by Yufei Gu <fl...@gmail.com>.
Could you help me understand the security concern? Every library can have
CVEs. Do you mean that the H2 library currently has unresolved CVEs, or
that it has historically had more CVEs than other libraries?
Yufei
On Fri, Jul 17, 2026 at 11:18 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi Yufei,
>
> What is your take on the security concerns associated with including the H2
> JDBC driver in the standard Polaris images (highlighted by Alex in this
> thread on Jul 3) [1]?
>
> [1] https://lists.apache.org/thread/8v42g3rvmt74b9s6ccgy7y8kky2hfbjo
>
> Thanks,
> Dmitri.
>
> On Fri, Jul 17, 2026 at 2:06 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > Hi Alex,
> >
> > Thanks for summarizing the discussion and separating the two questions. I
> > think that helps move us toward consensus.
> >
> > First of all, I agree with the direction, but I'm not sure it's urgent.
> > TreeMapMetaStore isn't perfect, but we already have both JDBC + H2 and
> > NoSQL + in-memory in place. That should already satisfy most of test
> > coverage.
> >
> > For question 1, I'd prefer Option 1a (JDBC + H2).
> >
> > My main reason is that it exercises the same persistence layer,
> transaction
> > handling, schema management, and configuration that most production
> > deployments use. That gives us a better out of the box experience for
> both
> > users and developers, while also increasing confidence that what works
> > locally will behave similarly in production.
> >
> > Do we have to wait for either PR 4812 or 4984? I think bundling the H2
> > driver should just unblock 1a.
> >
> > For question 2, I agree with handling test migration on a case by case
> > basis. Different tests have different goals, and using the most
> appropriate
> > backend for each test seems reasonable.
> >
> > Thanks,
> >
> > Yufei
> >
> >
> > On Fri, Jul 17, 2026 at 10:45 AM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Alex,
> > >
> > > Option 1b certainly makes sense to me. I support going in that
> direction.
> > >
> > > I'd propose opening a PR for that merging it before proceeding to test
> > code
> > > changes.
> > >
> > > In general, if 1b is acceptable, I'd think H2 should be removed [1]
> from
> > > tests and replaced with PostgreSQL or NoSQL InMemory. With that in
> mind,
> > > updating tests on a case-by-case basis seems reasonable to me. We can
> > > decide separately in each PR.
> > >
> > > [1] https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > >
> > > On Fri, Jul 17, 2026 at 12:59 PM Alexandre Dutra <ad...@apache.org>
> > > wrote:
> > >
> > > > Hi Dmitri, hi all,
> > > >
> > > > Unfortunately I'm not sure your interpretation matches mine, so let's
> > > > try to wrap up this discussion methodically.
> > > >
> > > > We agreed to deprecate TreeMapMetaStore for removal. But we haven't
> > > > reached consensus on two questions:
> > > >
> > > > 1) What should be the default metastore in the server image?
> > > > 2) How to replace TreeMapMetaStore in tests?
> > > >
> > > > Question 1) is tricky and needs a clear decision. We have only two
> > > > options for a smooth UX (aka, "docker run apache/polaris" just
> works):
> > > >
> > > > - Option 1a: JDBC+H2
> > > > - Option 1b: NoSQL+InMemory
> > > >
> > > > Option 1a is hard because it requires either my work on [4812]
> > > > (runtime activated datasources) or Yufei's work on [4984] (unmanaged
> > > > datasources). It also *requires* the H2 driver to be bundled by
> > > > default, which some people are not comfortable with. In short, I
> don't
> > > > see this option being feasible any time soon.
> > > >
> > > > Therefore, I think option 1b is much easier, and that would be my
> > > > preference.
> > > >
> > > > Question 2) is not as urgent, as I'm fairly sure we can find a
> > > > solution as we go. I can see a few options:
> > > > - Option 2a: migrate to JDBC+H2
> > > > - Option 2b: migrate to NoSQL+InMemory
> > > > - Option 2c: decide on a case-by-case basis
> > > >
> > > > My personal preference would be 2c.
> > > >
> > > > Can we try to reach consensus on the two questions please?
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > > [4984]: https://github.com/apache/polaris/pull/4984
> > > >
> > > > On Mon, Jul 6, 2026 at 8:29 PM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > > wrote:
> > > > >
> > > > > Hi All,
> > > > >
> > > > > My interpretation of the discussion so far is that:
> > > > >
> > > > > 1) We do want to deprecate the TreeMap MetaStore for removal
> > > > >
> > > > > 2) JDBC + PostgreSQL should be used for testing as well as NoSQL +
> > > > MongoDB
> > > > > (I believe it is in fact the case in current CI).
> > > > >
> > > > > 3) Instead of the TreeMap MetaStore, we will use the in-memory
> NoSQL
> > > > > MetaStore (for lightweight tests)
> > > > >
> > > > > Now, H2 becomes irrevant, IMHO, but removing it is being discussed
> > > > > separately [1].
> > > > >
> > > > > How does that sound?
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Fri, Jul 3, 2026 at 5:25 AM Alexandre Dutra <ad...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'm confused: security concerns were previously raised here [1]
> and
> > > > > > here [2] about bundling the H2 driver due to its past record of
> > > > > > critical CVEs.
> > > > > >
> > > > > > As a result, I assumed the JDBC+H2 option had been abandoned. Has
> > > this
> > > > > > proposal been brought back into consideration?
> > > > > >
> > > > > > Thanks,
> > > > > > Alex
> > > > > >
> > > > > > [1]:
> > > https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
> > > > > > [2]:
> > > https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 3, 2026 at 8:54 AM Jean-Baptiste Onofré <
> > jb@nanthrax.net
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > Yeah fully agree and it was my point when I replied on the
> thread
> > > > > > > initially.
> > > > > > >
> > > > > > > Regards
> > > > > > > JB
> > > > > > >
> > > > > > > Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <fl...@gmail.com>
> a
> > > > écrit :
> > > > > > >
> > > > > > > > If our goal is to make the default experience closer to
> > > production
> > > > > > > > while removing TreeMap, then JDBC with H2 seems like a more
> > > natural
> > > > > > > > choice. It exercises the same persistence layer, schema
> > > management,
> > > > > > > > and configuration that users will encounter in production,
> > while
> > > > still
> > > > > > > > keeping setup lightweight.
> > > > > > > >
> > > > > > > > Yufei
> > > > > > > >
> > > > > > > > Yufei
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <
> > > adutra@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Russell,
> > > > > > > > >
> > > > > > > > > The existing default metastore shouldn't be viewed merely
> as
> > a
> > > > > > > > > "relatively simple, self-contained implementation": it has
> > > unique
> > > > > > > > > quirks, such as returning result types that no other
> > metastore
> > > > does.
> > > > > > > > >
> > > > > > > > > Choosing the status quo simply to avoid making changes is
> > imho
> > > > the
> > > > > > > > > worst solution.
> > > > > > > > >
> > > > > > > > > Transitioning to the NoSQL metastore as the default
> remains a
> > > net
> > > > > > > > > benefit, even if it is only used by 30% of the user base.
> > > > > > > > >
> > > > > > > > > However, I agree with keeping the test suite on JDBC + H2
> by
> > > > default
> > > > > > > > > if that ensures a higher level of confidence in our tests
> > > > ability to
> > > > > > > > > reproduce real production setups.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Alex
> > > > > > > > >
> > > > > > > > > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
> > > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > I agree TreeMap and its associated transactional stack
> > should
> > > > be
> > > > > > > > deprecated.
> > > > > > > > > > It’s test-only, confusing, and not used by JDBC
> production
> > > > paths.
> > > > > > > > > >
> > > > > > > > > > I’m less convinced that NoSQL + InMemory is a clearly
> > > superior
> > > > > > > > replacement
> > > > > > > > > > for TreeMap in tests. At the storage layer it’s
> essentially
> > > the
> > > > > > same
> > > > > > > > class
> > > > > > > > > > of
> > > > > > > > > > thing: an in-memory, non-durable backend. The difference
> > is
> > > > that
> > > > > > it
> > > > > > > > > > exercises
> > > > > > > > > > a completely different MetaStoreManager/persistence stack
> > > > > > > > > > (NoSqlMetaStoreManager
> > > > > > > > > > vs TransactionalMetaStoreManagerImpl), not the JDBC path
> > most
> > > > > > > > deployments
> > > > > > > > > > use. It
> > > > > > > > > > replaces a relatively simple, self-contained
> implementation
> > > > with
> > > > > > > > something
> > > > > > > > > > that is
> > > > > > > > > > part of a much more complicated persistence stack, while
> > > > remaining
> > > > > > > > > > non-durable
> > > > > > > > > > and test-only at the storage layer.
> > > > > > > > > >
> > > > > > > > > > For the problems originally raised (test-grade defaults,
> > dead
> > > > > > > > production
> > > > > > > > > > path, developer confusion),
> > > > > > > > > > JDBC + H2 seems like an actual fix.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Alex,
> > > > > > > > > > >
> > > > > > > > > > > I tend to agree that the DataSource discussion related
> to
> > > H2
> > > > > > became
> > > > > > > > a bit
> > > > > > > > > > > convoluted, and it's a lot simpler to use the NoSQL
> > > in-memory
> > > > > > > > persistence
> > > > > > > > > > > instead.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <
> > > > > > adutra@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > Some updates on this:
> > > > > > > > > > > >
> > > > > > > > > > > > I recently started a separate discussion regarding
> > > dynamic
> > > > > > > > datasource
> > > > > > > > > > > > activation at runtime [1]. This was intended as a
> > > > prerequisite
> > > > > > for
> > > > > > > > > > > > adopting H2 as the default JDBC driver, as suggested
> on
> > > > this
> > > > > > > > thread.
> > > > > > > > > > > >
> > > > > > > > > > > > However, that proposal may end up being rejected.
> > > > > > > > > > > >
> > > > > > > > > > > > Given this situation, we should reconsider our
> options.
> > > If
> > > > the
> > > > > > > > > > > > PostgreSQL + H2 approach is no longer a viable
> > > replacement
> > > > to
> > > > > > the
> > > > > > > > > > > > TreeMapMetaStore, I propose an alternative: let's use
> > the
> > > > NoSQL
> > > > > > > > > > > > metastore with an InMemory backend.
> > > > > > > > > > > >
> > > > > > > > > > > > My reasoning is as follows:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. It remains a production-ready metastore, so
> strictly
> > > > > > superior to
> > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. The metastore state update logic is the same
> > > regardless
> > > > of
> > > > > > the
> > > > > > > > > > > > backend, so we're still close to a real production
> > setup
> > > > (same
> > > > > > as
> > > > > > > > with
> > > > > > > > > > > > an in-memory JDBC driver).
> > > > > > > > > > > >
> > > > > > > > > > > > 3. It doesn't require any external dependencies (no
> > > bundled
> > > > > > > > driver).
> > > > > > > > > > > >
> > > > > > > > > > > > I'd also note that while the NoSQL metastore is
> already
> > > > > > included
> > > > > > > > > > > > today, it is not actionable because no backend is
> > > > available.
> > > > > > This
> > > > > > > > goes
> > > > > > > > > > > > against the onboarding UX that we're trying to
> achieve,
> > > as
> > > > > > users
> > > > > > > > > > > > interested in Polaris with the NoSQL metastore cannot
> > > test
> > > > this
> > > > > > > > setup
> > > > > > > > > > > > with the official image.
> > > > > > > > > > > >
> > > > > > > > > > > > Therefore, I think NoSQL + InMemory would serve as a
> > more
> > > > > > > > appropriate
> > > > > > > > > > > > default setup for the official Polaris image.
> > > > > > > > > > > >
> > > > > > > > > > > > From what I see, it would boil down to adding one
> line
> > to
> > > > > > > > > > > > polaris-server or polaris-service:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > > > > > > > > > > >
> > > > > > > > > > > > What are your thoughts on this?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Alex
> > > > > > > > > > > >
> > > > > > > > > > > > [1]:
> > > > > > > >
> > https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra <
> > > > > > > > adutra@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I wanted to give an update on this topic: I have
> > > started
> > > > > > working
> > > > > > > > on
> > > > > > > > > > > > > this, but I would like to have [4812] merged first,
> > > since
> > > > > > that's
> > > > > > > > a
> > > > > > > > > > > > > prerequisite.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Alex
> > > > > > > > > > > > >
> > > > > > > > > > > > > [4812]:
> https://github.com/apache/polaris/pull/4812
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> > > > > > > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I believe the original idea of TreeMapMetastore
> was
> > > > that it
> > > > > > > > looked
> > > > > > > > > > > very
> > > > > > > > > > > > > > similar to FoundationDB from an API perspective,
> > so
> > > it
> > > > > > served
> > > > > > > > well
> > > > > > > > > > > as
> > > > > > > > > > > > a
> > > > > > > > > > > > > > test system for the original backend being
> > developed
> > > > at SF.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I agree that H2 + JDBC makes sense to me now for
> > the
> > > > > > project.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra
> <
> > > > > > > > adutra@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you for your feedback.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There seems to be a general agreement on the
> idea
> > > of
> > > > > > > > deprecating
> > > > > > > > > > > > > > > TreeMapMetaStore, coupled with the JDBC + H2
> > > > solution for
> > > > > > > > tests.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In most modules, the tests migration won't pose
> > any
> > > > > > serious
> > > > > > > > > > > > > > > challenges, as replacing the metastore is
> > generally
> > > > just
> > > > > > a
> > > > > > > > matter
> > > > > > > > > > > of
> > > > > > > > > > > > > > > changing the configuration, and making sure the
> > > > realm is
> > > > > > > > properly
> > > > > > > > > > > > > > > bootstrapped.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There will be, however, a few tricky situations
> > in
> > > > > > > > polaris-core: a
> > > > > > > > > > > > few
> > > > > > > > > > > > > > > tests rely on the TreeMapMetaStore, mostly as a
> > > test
> > > > > > > > convenience;
> > > > > > > > > > > but
> > > > > > > > > > > > > > > there is no obvious replacement for it in that
> > > > module. I
> > > > > > am
> > > > > > > > however
> > > > > > > > > > > > > > > confident that we can find a solution for that,
> > > > either
> > > > > > based
> > > > > > > > on
> > > > > > > > > > > > mocks,
> > > > > > > > > > > > > > > or by bringing in a real metastore.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If no objections are raised, I am going to
> > prepare
> > > a
> > > > PR
> > > > > > for
> > > > > > > > this.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri
> > > Bourlatchkov <
> > > > > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for starting this thread.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I also find the
> > TransactionalMetaStoreManagerImpl
> > > > and
> > > > > > > > related
> > > > > > > > > > > call
> > > > > > > > > > > > paths
> > > > > > > > > > > > > > > > conuising in Apache Polaris code.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I support promoting H2 to the default (in
> > memory)
> > > > > > > > Persistence
> > > > > > > > > > > > backend for
> > > > > > > > > > > > > > > > getting started cases. This should also
> resolve
> > > the
> > > > > > old H2
> > > > > > > > > > > > evolution
> > > > > > > > > > > > > > > thread
> > > > > > > > > > > > > > > > [1] by ensuring it is used regularly on the
> > same
> > > > code
> > > > > > > > paths as
> > > > > > > > > > > > > > > PostgreSQL.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Deprecating TransactionalMetaStoreManagerImpl
> > for
> > > > > > removal
> > > > > > > > also
> > > > > > > > > > > > sounds
> > > > > > > > > > > > > > > > reasonable to me. Existing downstream users
> > > > > > > > > > > > > > > > of TransactionalMetaStoreManagerImpl will
> have
> > > > time to
> > > > > > > > migrate,
> > > > > > > > > > > or
> > > > > > > > > > > > even
> > > > > > > > > > > > > > > > embed that code (per ASF license) into local
> > > > builds,
> > > > > > > > during the
> > > > > > > > > > > > > > > deprecation
> > > > > > > > > > > > > > > > phase.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > >
> > > > > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre
> > Dutra
> > > <
> > > > > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I am writing to ask the community whether
> it
> > is
> > > > OK to
> > > > > > > > deprecate
> > > > > > > > > > > > > > > > > TreeMapMetaStore, as well as all the
> > in-memory
> > > > > > metastore
> > > > > > > > > > > manager,
> > > > > > > > > > > > > > > > > metastore manager factory, and persistence
> > > types
> > > > that
> > > > > > > > rely
> > > > > > > > > > > > solely on
> > > > > > > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > As we know, these components are test-grade
> > > only,
> > > > > > and not
> > > > > > > > > > > > suitable for
> > > > > > > > > > > > > > > > > production. They trigger a production
> > readiness
> > > > > > alert on
> > > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > > startup. It's a considerable amount of code
> > > that
> > > > is
> > > > > > > > virtually
> > > > > > > > > > > > dead in
> > > > > > > > > > > > > > > > > production.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It's also confusing for developers. E.g.
> the
> > > > > > > > "transactional"
> > > > > > > > > > > > metastore
> > > > > > > > > > > > > > > > > is not transactional in the JDBC sense of
> the
> > > > term,
> > > > > > and
> > > > > > > > thus
> > > > > > > > > > > not
> > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > by JDBC persistence. It also has its
> quirks:
> > > some
> > > > > > return
> > > > > > > > > > > > statuses are
> > > > > > > > > > > > > > > > > only returned by that manager.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > However, these components are used in
> tests,
> > > and
> > > > I
> > > > > > agree
> > > > > > > > that
> > > > > > > > > > > > it's
> > > > > > > > > > > > > > > > > useful to have an in-memory version of the
> > > > > > persistence
> > > > > > > > layer
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > > tests. But we have today two alternatives
> > that
> > > > are
> > > > > > imho
> > > > > > > > > > > superior
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > tests in polaris-runtime-service:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > - JDBC persistence with H2 backend. There
> are
> > > > > > already a
> > > > > > > > few
> > > > > > > > > > > tests
> > > > > > > > > > > > > > > > > using this setup.
> > > > > > > > > > > > > > > > > - NoSQL persistence with InMemory backend.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Both alternatives test real
> production-grade
> > > > > > persistence
> > > > > > > > code.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > And finally, TreeMapMetaStore is currently
> > the
> > > > > > default
> > > > > > > > runtime
> > > > > > > > > > > > > > > > > persistence in application.properties; and
> > the
> > > > Helm
> > > > > > > > chart also
> > > > > > > > > > > > > > > > > advertises it as the default. These are not
> > > sane
> > > > > > > > defaults,
> > > > > > > > > > > imho.
> > > > > > > > > > > > It's
> > > > > > > > > > > > > > > > > always tricky to provide a good default for
> > > > > > datastores,
> > > > > > > > but
> > > > > > > > > > > > since JDBC
> > > > > > > > > > > > > > > > > persistence is included by default in the
> > > server
> > > > > > image,
> > > > > > > > I think
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > including the H2 driver by default could
> give
> > > us
> > > > a
> > > > > > saner
> > > > > > > > > > > default
> > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > keeping the out-of-the-box experience
> intact
> > > (the
> > > > > > > > license is
> > > > > > > > > > > > Category
> > > > > > > > > > > > > > > > > A).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Concretely:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On the MetaStoreManagerFactory hierarchy,
> the
> > > > > > following
> > > > > > > > > > > > > > > > > implementations are completely in-memory:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > - InMemoryPolarisMetaStoreManagerFactory:
> > could
> > > > be
> > > > > > > > removed
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > -
> > > InMemoryAtomicOperationMetaStoreManagerFactory:
> > > > > > could
> > > > > > > > be
> > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > - LocalPolarisMetaStoreManagerFactory: is
> the
> > > > base
> > > > > > class
> > > > > > > > of the
> > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > above; imho it can be removed, but since
> it's
> > > an
> > > > > > abstract
> > > > > > > > > > > class,
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > may have been extended outside Polaris. But
> > > > neither
> > > > > > JDBC
> > > > > > > > nor
> > > > > > > > > > > > NoSQL use
> > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On the PolarisMetaStoreManager hierarchy:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > - TransactionalMetaStoreManagerImpl: is
> only
> > > > used by
> > > > > > > > > > > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC
> and
> > > > NoSQL
> > > > > > do
> > > > > > > > not use
> > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > Could be removed.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > - TransactionWorkspaceMetaStoreManager
> > however
> > > > is a
> > > > > > > > different
> > > > > > > > > > > > beast,
> > > > > > > > > > > > > > > > > in spite of the similar name. It is in use
> > > today
> > > > on
> > > > > > the
> > > > > > > > commit
> > > > > > > > > > > > path,
> > > > > > > > > > > > > > > > > so should not be removed.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On the BasePersistence hierarchy:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > - TreeMapTransactionalPersistenceImpl and
> its
> > > > > > > > TreeMapMetaStore
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > only used by
> > > > InMemoryPolarisMetaStoreManagerFactory,
> > > > > > and
> > > > > > > > could
> > > > > > > > > > > be
> > > > > > > > > > > > > > > > > removed;
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > - TransactionalPersistence and
> > > > > > > > > > > AbstractTransactionalPersistence:
> > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > are supertypes of
> > > > TreeMapTransactionalPersistenceImpl
> > > > > > > > and thus
> > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > used for in-memory. They imo can be
> removed,
> > > but
> > > > they
> > > > > > > > might
> > > > > > > > > > > have
> > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > extended or implemented outside Polaris.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think it's important to keep Polaris code
> > > tidy
> > > > by
> > > > > > > > removing
> > > > > > > > > > > > unused,
> > > > > > > > > > > > > > > > > unimplementable, or test-grade only
> > components.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > What are your thoughts?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Yufei,
What is your take on the security concerns associated with including the H2
JDBC driver in the standard Polaris images (highlighted by Alex in this
thread on Jul 3) [1]?
[1] https://lists.apache.org/thread/8v42g3rvmt74b9s6ccgy7y8kky2hfbjo
Thanks,
Dmitri.
On Fri, Jul 17, 2026 at 2:06 PM Yufei Gu <fl...@gmail.com> wrote:
> Hi Alex,
>
> Thanks for summarizing the discussion and separating the two questions. I
> think that helps move us toward consensus.
>
> First of all, I agree with the direction, but I'm not sure it's urgent.
> TreeMapMetaStore isn't perfect, but we already have both JDBC + H2 and
> NoSQL + in-memory in place. That should already satisfy most of test
> coverage.
>
> For question 1, I'd prefer Option 1a (JDBC + H2).
>
> My main reason is that it exercises the same persistence layer, transaction
> handling, schema management, and configuration that most production
> deployments use. That gives us a better out of the box experience for both
> users and developers, while also increasing confidence that what works
> locally will behave similarly in production.
>
> Do we have to wait for either PR 4812 or 4984? I think bundling the H2
> driver should just unblock 1a.
>
> For question 2, I agree with handling test migration on a case by case
> basis. Different tests have different goals, and using the most appropriate
> backend for each test seems reasonable.
>
> Thanks,
>
> Yufei
>
>
> On Fri, Jul 17, 2026 at 10:45 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Alex,
> >
> > Option 1b certainly makes sense to me. I support going in that direction.
> >
> > I'd propose opening a PR for that merging it before proceeding to test
> code
> > changes.
> >
> > In general, if 1b is acceptable, I'd think H2 should be removed [1] from
> > tests and replaced with PostgreSQL or NoSQL InMemory. With that in mind,
> > updating tests on a case-by-case basis seems reasonable to me. We can
> > decide separately in each PR.
> >
> > [1] https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> >
> > Cheers,
> > Dmitri.
> >
> >
> > On Fri, Jul 17, 2026 at 12:59 PM Alexandre Dutra <ad...@apache.org>
> > wrote:
> >
> > > Hi Dmitri, hi all,
> > >
> > > Unfortunately I'm not sure your interpretation matches mine, so let's
> > > try to wrap up this discussion methodically.
> > >
> > > We agreed to deprecate TreeMapMetaStore for removal. But we haven't
> > > reached consensus on two questions:
> > >
> > > 1) What should be the default metastore in the server image?
> > > 2) How to replace TreeMapMetaStore in tests?
> > >
> > > Question 1) is tricky and needs a clear decision. We have only two
> > > options for a smooth UX (aka, "docker run apache/polaris" just works):
> > >
> > > - Option 1a: JDBC+H2
> > > - Option 1b: NoSQL+InMemory
> > >
> > > Option 1a is hard because it requires either my work on [4812]
> > > (runtime activated datasources) or Yufei's work on [4984] (unmanaged
> > > datasources). It also *requires* the H2 driver to be bundled by
> > > default, which some people are not comfortable with. In short, I don't
> > > see this option being feasible any time soon.
> > >
> > > Therefore, I think option 1b is much easier, and that would be my
> > > preference.
> > >
> > > Question 2) is not as urgent, as I'm fairly sure we can find a
> > > solution as we go. I can see a few options:
> > > - Option 2a: migrate to JDBC+H2
> > > - Option 2b: migrate to NoSQL+InMemory
> > > - Option 2c: decide on a case-by-case basis
> > >
> > > My personal preference would be 2c.
> > >
> > > Can we try to reach consensus on the two questions please?
> > >
> > > Thanks,
> > > Alex
> > >
> > > [4812]: https://github.com/apache/polaris/pull/4812
> > > [4984]: https://github.com/apache/polaris/pull/4984
> > >
> > > On Mon, Jul 6, 2026 at 8:29 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > > >
> > > > Hi All,
> > > >
> > > > My interpretation of the discussion so far is that:
> > > >
> > > > 1) We do want to deprecate the TreeMap MetaStore for removal
> > > >
> > > > 2) JDBC + PostgreSQL should be used for testing as well as NoSQL +
> > > MongoDB
> > > > (I believe it is in fact the case in current CI).
> > > >
> > > > 3) Instead of the TreeMap MetaStore, we will use the in-memory NoSQL
> > > > MetaStore (for lightweight tests)
> > > >
> > > > Now, H2 becomes irrevant, IMHO, but removing it is being discussed
> > > > separately [1].
> > > >
> > > > How does that sound?
> > > >
> > > > [1] https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Fri, Jul 3, 2026 at 5:25 AM Alexandre Dutra <ad...@apache.org>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm confused: security concerns were previously raised here [1] and
> > > > > here [2] about bundling the H2 driver due to its past record of
> > > > > critical CVEs.
> > > > >
> > > > > As a result, I assumed the JDBC+H2 option had been abandoned. Has
> > this
> > > > > proposal been brought back into consideration?
> > > > >
> > > > > Thanks,
> > > > > Alex
> > > > >
> > > > > [1]:
> > https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
> > > > > [2]:
> > https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
> > > > >
> > > > >
> > > > > On Fri, Jul 3, 2026 at 8:54 AM Jean-Baptiste Onofré <
> jb@nanthrax.net
> > >
> > > > > wrote:
> > > > > >
> > > > > > Hi
> > > > > >
> > > > > > Yeah fully agree and it was my point when I replied on the thread
> > > > > > initially.
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > > Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <fl...@gmail.com> a
> > > écrit :
> > > > > >
> > > > > > > If our goal is to make the default experience closer to
> > production
> > > > > > > while removing TreeMap, then JDBC with H2 seems like a more
> > natural
> > > > > > > choice. It exercises the same persistence layer, schema
> > management,
> > > > > > > and configuration that users will encounter in production,
> while
> > > still
> > > > > > > keeping setup lightweight.
> > > > > > >
> > > > > > > Yufei
> > > > > > >
> > > > > > > Yufei
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <
> > adutra@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Russell,
> > > > > > > >
> > > > > > > > The existing default metastore shouldn't be viewed merely as
> a
> > > > > > > > "relatively simple, self-contained implementation": it has
> > unique
> > > > > > > > quirks, such as returning result types that no other
> metastore
> > > does.
> > > > > > > >
> > > > > > > > Choosing the status quo simply to avoid making changes is
> imho
> > > the
> > > > > > > > worst solution.
> > > > > > > >
> > > > > > > > Transitioning to the NoSQL metastore as the default remains a
> > net
> > > > > > > > benefit, even if it is only used by 30% of the user base.
> > > > > > > >
> > > > > > > > However, I agree with keeping the test suite on JDBC + H2 by
> > > default
> > > > > > > > if that ensures a higher level of confidence in our tests
> > > ability to
> > > > > > > > reproduce real production setups.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Alex
> > > > > > > >
> > > > > > > > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
> > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > I agree TreeMap and its associated transactional stack
> should
> > > be
> > > > > > > deprecated.
> > > > > > > > > It’s test-only, confusing, and not used by JDBC production
> > > paths.
> > > > > > > > >
> > > > > > > > > I’m less convinced that NoSQL + InMemory is a clearly
> > superior
> > > > > > > replacement
> > > > > > > > > for TreeMap in tests. At the storage layer it’s essentially
> > the
> > > > > same
> > > > > > > class
> > > > > > > > > of
> > > > > > > > > thing: an in-memory, non-durable backend. The difference
> is
> > > that
> > > > > it
> > > > > > > > > exercises
> > > > > > > > > a completely different MetaStoreManager/persistence stack
> > > > > > > > > (NoSqlMetaStoreManager
> > > > > > > > > vs TransactionalMetaStoreManagerImpl), not the JDBC path
> most
> > > > > > > deployments
> > > > > > > > > use. It
> > > > > > > > > replaces a relatively simple, self-contained implementation
> > > with
> > > > > > > something
> > > > > > > > > that is
> > > > > > > > > part of a much more complicated persistence stack, while
> > > remaining
> > > > > > > > > non-durable
> > > > > > > > > and test-only at the storage layer.
> > > > > > > > >
> > > > > > > > > For the problems originally raised (test-grade defaults,
> dead
> > > > > > > production
> > > > > > > > > path, developer confusion),
> > > > > > > > > JDBC + H2 seems like an actual fix.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Alex,
> > > > > > > > > >
> > > > > > > > > > I tend to agree that the DataSource discussion related to
> > H2
> > > > > became
> > > > > > > a bit
> > > > > > > > > > convoluted, and it's a lot simpler to use the NoSQL
> > in-memory
> > > > > > > persistence
> > > > > > > > > > instead.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <
> > > > > adutra@apache.org>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > Some updates on this:
> > > > > > > > > > >
> > > > > > > > > > > I recently started a separate discussion regarding
> > dynamic
> > > > > > > datasource
> > > > > > > > > > > activation at runtime [1]. This was intended as a
> > > prerequisite
> > > > > for
> > > > > > > > > > > adopting H2 as the default JDBC driver, as suggested on
> > > this
> > > > > > > thread.
> > > > > > > > > > >
> > > > > > > > > > > However, that proposal may end up being rejected.
> > > > > > > > > > >
> > > > > > > > > > > Given this situation, we should reconsider our options.
> > If
> > > the
> > > > > > > > > > > PostgreSQL + H2 approach is no longer a viable
> > replacement
> > > to
> > > > > the
> > > > > > > > > > > TreeMapMetaStore, I propose an alternative: let's use
> the
> > > NoSQL
> > > > > > > > > > > metastore with an InMemory backend.
> > > > > > > > > > >
> > > > > > > > > > > My reasoning is as follows:
> > > > > > > > > > >
> > > > > > > > > > > 1. It remains a production-ready metastore, so strictly
> > > > > superior to
> > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > >
> > > > > > > > > > > 2. The metastore state update logic is the same
> > regardless
> > > of
> > > > > the
> > > > > > > > > > > backend, so we're still close to a real production
> setup
> > > (same
> > > > > as
> > > > > > > with
> > > > > > > > > > > an in-memory JDBC driver).
> > > > > > > > > > >
> > > > > > > > > > > 3. It doesn't require any external dependencies (no
> > bundled
> > > > > > > driver).
> > > > > > > > > > >
> > > > > > > > > > > I'd also note that while the NoSQL metastore is already
> > > > > included
> > > > > > > > > > > today, it is not actionable because no backend is
> > > available.
> > > > > This
> > > > > > > goes
> > > > > > > > > > > against the onboarding UX that we're trying to achieve,
> > as
> > > > > users
> > > > > > > > > > > interested in Polaris with the NoSQL metastore cannot
> > test
> > > this
> > > > > > > setup
> > > > > > > > > > > with the official image.
> > > > > > > > > > >
> > > > > > > > > > > Therefore, I think NoSQL + InMemory would serve as a
> more
> > > > > > > appropriate
> > > > > > > > > > > default setup for the official Polaris image.
> > > > > > > > > > >
> > > > > > > > > > > From what I see, it would boil down to adding one line
> to
> > > > > > > > > > > polaris-server or polaris-service:
> > > > > > > > > > >
> > > > > > > > > > >
> > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > > > > > > > > > >
> > > > > > > > > > > What are your thoughts on this?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Alex
> > > > > > > > > > >
> > > > > > > > > > > [1]:
> > > > > > >
> https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra <
> > > > > > > adutra@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I wanted to give an update on this topic: I have
> > started
> > > > > working
> > > > > > > on
> > > > > > > > > > > > this, but I would like to have [4812] merged first,
> > since
> > > > > that's
> > > > > > > a
> > > > > > > > > > > > prerequisite.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Alex
> > > > > > > > > > > >
> > > > > > > > > > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> > > > > > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > I believe the original idea of TreeMapMetastore was
> > > that it
> > > > > > > looked
> > > > > > > > > > very
> > > > > > > > > > > > > similar to FoundationDB from an API perspective,
> so
> > it
> > > > > served
> > > > > > > well
> > > > > > > > > > as
> > > > > > > > > > > a
> > > > > > > > > > > > > test system for the original backend being
> developed
> > > at SF.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I agree that H2 + JDBC makes sense to me now for
> the
> > > > > project.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra <
> > > > > > > adutra@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you for your feedback.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There seems to be a general agreement on the idea
> > of
> > > > > > > deprecating
> > > > > > > > > > > > > > TreeMapMetaStore, coupled with the JDBC + H2
> > > solution for
> > > > > > > tests.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In most modules, the tests migration won't pose
> any
> > > > > serious
> > > > > > > > > > > > > > challenges, as replacing the metastore is
> generally
> > > just
> > > > > a
> > > > > > > matter
> > > > > > > > > > of
> > > > > > > > > > > > > > changing the configuration, and making sure the
> > > realm is
> > > > > > > properly
> > > > > > > > > > > > > > bootstrapped.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There will be, however, a few tricky situations
> in
> > > > > > > polaris-core: a
> > > > > > > > > > > few
> > > > > > > > > > > > > > tests rely on the TreeMapMetaStore, mostly as a
> > test
> > > > > > > convenience;
> > > > > > > > > > but
> > > > > > > > > > > > > > there is no obvious replacement for it in that
> > > module. I
> > > > > am
> > > > > > > however
> > > > > > > > > > > > > > confident that we can find a solution for that,
> > > either
> > > > > based
> > > > > > > on
> > > > > > > > > > > mocks,
> > > > > > > > > > > > > > or by bringing in a real metastore.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If no objections are raised, I am going to
> prepare
> > a
> > > PR
> > > > > for
> > > > > > > this.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri
> > Bourlatchkov <
> > > > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for starting this thread.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I also find the
> TransactionalMetaStoreManagerImpl
> > > and
> > > > > > > related
> > > > > > > > > > call
> > > > > > > > > > > paths
> > > > > > > > > > > > > > > conuising in Apache Polaris code.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I support promoting H2 to the default (in
> memory)
> > > > > > > Persistence
> > > > > > > > > > > backend for
> > > > > > > > > > > > > > > getting started cases. This should also resolve
> > the
> > > > > old H2
> > > > > > > > > > > evolution
> > > > > > > > > > > > > > thread
> > > > > > > > > > > > > > > [1] by ensuring it is used regularly on the
> same
> > > code
> > > > > > > paths as
> > > > > > > > > > > > > > PostgreSQL.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Deprecating TransactionalMetaStoreManagerImpl
> for
> > > > > removal
> > > > > > > also
> > > > > > > > > > > sounds
> > > > > > > > > > > > > > > reasonable to me. Existing downstream users
> > > > > > > > > > > > > > > of TransactionalMetaStoreManagerImpl will have
> > > time to
> > > > > > > migrate,
> > > > > > > > > > or
> > > > > > > > > > > even
> > > > > > > > > > > > > > > embed that code (per ASF license) into local
> > > builds,
> > > > > > > during the
> > > > > > > > > > > > > > deprecation
> > > > > > > > > > > > > > > phase.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre
> Dutra
> > <
> > > > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I am writing to ask the community whether it
> is
> > > OK to
> > > > > > > deprecate
> > > > > > > > > > > > > > > > TreeMapMetaStore, as well as all the
> in-memory
> > > > > metastore
> > > > > > > > > > manager,
> > > > > > > > > > > > > > > > metastore manager factory, and persistence
> > types
> > > that
> > > > > > > rely
> > > > > > > > > > > solely on
> > > > > > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > As we know, these components are test-grade
> > only,
> > > > > and not
> > > > > > > > > > > suitable for
> > > > > > > > > > > > > > > > production. They trigger a production
> readiness
> > > > > alert on
> > > > > > > > > > Polaris
> > > > > > > > > > > > > > > > startup. It's a considerable amount of code
> > that
> > > is
> > > > > > > virtually
> > > > > > > > > > > dead in
> > > > > > > > > > > > > > > > production.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It's also confusing for developers. E.g. the
> > > > > > > "transactional"
> > > > > > > > > > > metastore
> > > > > > > > > > > > > > > > is not transactional in the JDBC sense of the
> > > term,
> > > > > and
> > > > > > > thus
> > > > > > > > > > not
> > > > > > > > > > > used
> > > > > > > > > > > > > > > > by JDBC persistence. It also has its quirks:
> > some
> > > > > return
> > > > > > > > > > > statuses are
> > > > > > > > > > > > > > > > only returned by that manager.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > However, these components are used in tests,
> > and
> > > I
> > > > > agree
> > > > > > > that
> > > > > > > > > > > it's
> > > > > > > > > > > > > > > > useful to have an in-memory version of the
> > > > > persistence
> > > > > > > layer
> > > > > > > > > > for
> > > > > > > > > > > > > > > > tests. But we have today two alternatives
> that
> > > are
> > > > > imho
> > > > > > > > > > superior
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > tests in polaris-runtime-service:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - JDBC persistence with H2 backend. There are
> > > > > already a
> > > > > > > few
> > > > > > > > > > tests
> > > > > > > > > > > > > > > > using this setup.
> > > > > > > > > > > > > > > > - NoSQL persistence with InMemory backend.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Both alternatives test real production-grade
> > > > > persistence
> > > > > > > code.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > And finally, TreeMapMetaStore is currently
> the
> > > > > default
> > > > > > > runtime
> > > > > > > > > > > > > > > > persistence in application.properties; and
> the
> > > Helm
> > > > > > > chart also
> > > > > > > > > > > > > > > > advertises it as the default. These are not
> > sane
> > > > > > > defaults,
> > > > > > > > > > imho.
> > > > > > > > > > > It's
> > > > > > > > > > > > > > > > always tricky to provide a good default for
> > > > > datastores,
> > > > > > > but
> > > > > > > > > > > since JDBC
> > > > > > > > > > > > > > > > persistence is included by default in the
> > server
> > > > > image,
> > > > > > > I think
> > > > > > > > > > > that
> > > > > > > > > > > > > > > > including the H2 driver by default could give
> > us
> > > a
> > > > > saner
> > > > > > > > > > default
> > > > > > > > > > > while
> > > > > > > > > > > > > > > > keeping the out-of-the-box experience intact
> > (the
> > > > > > > license is
> > > > > > > > > > > Category
> > > > > > > > > > > > > > > > A).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Concretely:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On the MetaStoreManagerFactory hierarchy, the
> > > > > following
> > > > > > > > > > > > > > > > implementations are completely in-memory:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - InMemoryPolarisMetaStoreManagerFactory:
> could
> > > be
> > > > > > > removed
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > -
> > InMemoryAtomicOperationMetaStoreManagerFactory:
> > > > > could
> > > > > > > be
> > > > > > > > > > > removed
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - LocalPolarisMetaStoreManagerFactory: is the
> > > base
> > > > > class
> > > > > > > of the
> > > > > > > > > > > two
> > > > > > > > > > > > > > > > above; imho it can be removed, but since it's
> > an
> > > > > abstract
> > > > > > > > > > class,
> > > > > > > > > > > it
> > > > > > > > > > > > > > > > may have been extended outside Polaris. But
> > > neither
> > > > > JDBC
> > > > > > > nor
> > > > > > > > > > > NoSQL use
> > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On the PolarisMetaStoreManager hierarchy:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - TransactionalMetaStoreManagerImpl: is only
> > > used by
> > > > > > > > > > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC and
> > > NoSQL
> > > > > do
> > > > > > > not use
> > > > > > > > > > > it.
> > > > > > > > > > > > > > > > Could be removed.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - TransactionWorkspaceMetaStoreManager
> however
> > > is a
> > > > > > > different
> > > > > > > > > > > beast,
> > > > > > > > > > > > > > > > in spite of the similar name. It is in use
> > today
> > > on
> > > > > the
> > > > > > > commit
> > > > > > > > > > > path,
> > > > > > > > > > > > > > > > so should not be removed.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On the BasePersistence hierarchy:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - TreeMapTransactionalPersistenceImpl and its
> > > > > > > TreeMapMetaStore
> > > > > > > > > > > are
> > > > > > > > > > > > > > > > only used by
> > > InMemoryPolarisMetaStoreManagerFactory,
> > > > > and
> > > > > > > could
> > > > > > > > > > be
> > > > > > > > > > > > > > > > removed;
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - TransactionalPersistence and
> > > > > > > > > > AbstractTransactionalPersistence:
> > > > > > > > > > > these
> > > > > > > > > > > > > > > > are supertypes of
> > > TreeMapTransactionalPersistenceImpl
> > > > > > > and thus
> > > > > > > > > > > only
> > > > > > > > > > > > > > > > used for in-memory. They imo can be removed,
> > but
> > > they
> > > > > > > might
> > > > > > > > > > have
> > > > > > > > > > > been
> > > > > > > > > > > > > > > > extended or implemented outside Polaris.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think it's important to keep Polaris code
> > tidy
> > > by
> > > > > > > removing
> > > > > > > > > > > unused,
> > > > > > > > > > > > > > > > unimplementable, or test-grade only
> components.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > What are your thoughts?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > >
> > >
> >
>
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Posted by Yufei Gu <fl...@gmail.com>.
Hi Alex,
Thanks for summarizing the discussion and separating the two questions. I
think that helps move us toward consensus.
First of all, I agree with the direction, but I'm not sure it's urgent.
TreeMapMetaStore isn't perfect, but we already have both JDBC + H2 and
NoSQL + in-memory in place. That should already satisfy most of test
coverage.
For question 1, I'd prefer Option 1a (JDBC + H2).
My main reason is that it exercises the same persistence layer, transaction
handling, schema management, and configuration that most production
deployments use. That gives us a better out of the box experience for both
users and developers, while also increasing confidence that what works
locally will behave similarly in production.
Do we have to wait for either PR 4812 or 4984? I think bundling the H2
driver should just unblock 1a.
For question 2, I agree with handling test migration on a case by case
basis. Different tests have different goals, and using the most appropriate
backend for each test seems reasonable.
Thanks,
Yufei
On Fri, Jul 17, 2026 at 10:45 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi Alex,
>
> Option 1b certainly makes sense to me. I support going in that direction.
>
> I'd propose opening a PR for that merging it before proceeding to test code
> changes.
>
> In general, if 1b is acceptable, I'd think H2 should be removed [1] from
> tests and replaced with PostgreSQL or NoSQL InMemory. With that in mind,
> updating tests on a case-by-case basis seems reasonable to me. We can
> decide separately in each PR.
>
> [1] https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
>
> Cheers,
> Dmitri.
>
>
> On Fri, Jul 17, 2026 at 12:59 PM Alexandre Dutra <ad...@apache.org>
> wrote:
>
> > Hi Dmitri, hi all,
> >
> > Unfortunately I'm not sure your interpretation matches mine, so let's
> > try to wrap up this discussion methodically.
> >
> > We agreed to deprecate TreeMapMetaStore for removal. But we haven't
> > reached consensus on two questions:
> >
> > 1) What should be the default metastore in the server image?
> > 2) How to replace TreeMapMetaStore in tests?
> >
> > Question 1) is tricky and needs a clear decision. We have only two
> > options for a smooth UX (aka, "docker run apache/polaris" just works):
> >
> > - Option 1a: JDBC+H2
> > - Option 1b: NoSQL+InMemory
> >
> > Option 1a is hard because it requires either my work on [4812]
> > (runtime activated datasources) or Yufei's work on [4984] (unmanaged
> > datasources). It also *requires* the H2 driver to be bundled by
> > default, which some people are not comfortable with. In short, I don't
> > see this option being feasible any time soon.
> >
> > Therefore, I think option 1b is much easier, and that would be my
> > preference.
> >
> > Question 2) is not as urgent, as I'm fairly sure we can find a
> > solution as we go. I can see a few options:
> > - Option 2a: migrate to JDBC+H2
> > - Option 2b: migrate to NoSQL+InMemory
> > - Option 2c: decide on a case-by-case basis
> >
> > My personal preference would be 2c.
> >
> > Can we try to reach consensus on the two questions please?
> >
> > Thanks,
> > Alex
> >
> > [4812]: https://github.com/apache/polaris/pull/4812
> > [4984]: https://github.com/apache/polaris/pull/4984
> >
> > On Mon, Jul 6, 2026 at 8:29 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> > >
> > > Hi All,
> > >
> > > My interpretation of the discussion so far is that:
> > >
> > > 1) We do want to deprecate the TreeMap MetaStore for removal
> > >
> > > 2) JDBC + PostgreSQL should be used for testing as well as NoSQL +
> > MongoDB
> > > (I believe it is in fact the case in current CI).
> > >
> > > 3) Instead of the TreeMap MetaStore, we will use the in-memory NoSQL
> > > MetaStore (for lightweight tests)
> > >
> > > Now, H2 becomes irrevant, IMHO, but removing it is being discussed
> > > separately [1].
> > >
> > > How does that sound?
> > >
> > > [1] https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Fri, Jul 3, 2026 at 5:25 AM Alexandre Dutra <ad...@apache.org>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm confused: security concerns were previously raised here [1] and
> > > > here [2] about bundling the H2 driver due to its past record of
> > > > critical CVEs.
> > > >
> > > > As a result, I assumed the JDBC+H2 option had been abandoned. Has
> this
> > > > proposal been brought back into consideration?
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > > > [1]:
> https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
> > > > [2]:
> https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
> > > >
> > > >
> > > > On Fri, Jul 3, 2026 at 8:54 AM Jean-Baptiste Onofré <jb@nanthrax.net
> >
> > > > wrote:
> > > > >
> > > > > Hi
> > > > >
> > > > > Yeah fully agree and it was my point when I replied on the thread
> > > > > initially.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <fl...@gmail.com> a
> > écrit :
> > > > >
> > > > > > If our goal is to make the default experience closer to
> production
> > > > > > while removing TreeMap, then JDBC with H2 seems like a more
> natural
> > > > > > choice. It exercises the same persistence layer, schema
> management,
> > > > > > and configuration that users will encounter in production, while
> > still
> > > > > > keeping setup lightweight.
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <
> adutra@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > Hi Russell,
> > > > > > >
> > > > > > > The existing default metastore shouldn't be viewed merely as a
> > > > > > > "relatively simple, self-contained implementation": it has
> unique
> > > > > > > quirks, such as returning result types that no other metastore
> > does.
> > > > > > >
> > > > > > > Choosing the status quo simply to avoid making changes is imho
> > the
> > > > > > > worst solution.
> > > > > > >
> > > > > > > Transitioning to the NoSQL metastore as the default remains a
> net
> > > > > > > benefit, even if it is only used by 30% of the user base.
> > > > > > >
> > > > > > > However, I agree with keeping the test suite on JDBC + H2 by
> > default
> > > > > > > if that ensures a higher level of confidence in our tests
> > ability to
> > > > > > > reproduce real production setups.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Alex
> > > > > > >
> > > > > > > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
> > > > > > > <ru...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > I agree TreeMap and its associated transactional stack should
> > be
> > > > > > deprecated.
> > > > > > > > It’s test-only, confusing, and not used by JDBC production
> > paths.
> > > > > > > >
> > > > > > > > I’m less convinced that NoSQL + InMemory is a clearly
> superior
> > > > > > replacement
> > > > > > > > for TreeMap in tests. At the storage layer it’s essentially
> the
> > > > same
> > > > > > class
> > > > > > > > of
> > > > > > > > thing: an in-memory, non-durable backend. The difference is
> > that
> > > > it
> > > > > > > > exercises
> > > > > > > > a completely different MetaStoreManager/persistence stack
> > > > > > > > (NoSqlMetaStoreManager
> > > > > > > > vs TransactionalMetaStoreManagerImpl), not the JDBC path most
> > > > > > deployments
> > > > > > > > use. It
> > > > > > > > replaces a relatively simple, self-contained implementation
> > with
> > > > > > something
> > > > > > > > that is
> > > > > > > > part of a much more complicated persistence stack, while
> > remaining
> > > > > > > > non-durable
> > > > > > > > and test-only at the storage layer.
> > > > > > > >
> > > > > > > > For the problems originally raised (test-grade defaults, dead
> > > > > > production
> > > > > > > > path, developer confusion),
> > > > > > > > JDBC + H2 seems like an actual fix.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Alex,
> > > > > > > > >
> > > > > > > > > I tend to agree that the DataSource discussion related to
> H2
> > > > became
> > > > > > a bit
> > > > > > > > > convoluted, and it's a lot simpler to use the NoSQL
> in-memory
> > > > > > persistence
> > > > > > > > > instead.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <
> > > > adutra@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > Some updates on this:
> > > > > > > > > >
> > > > > > > > > > I recently started a separate discussion regarding
> dynamic
> > > > > > datasource
> > > > > > > > > > activation at runtime [1]. This was intended as a
> > prerequisite
> > > > for
> > > > > > > > > > adopting H2 as the default JDBC driver, as suggested on
> > this
> > > > > > thread.
> > > > > > > > > >
> > > > > > > > > > However, that proposal may end up being rejected.
> > > > > > > > > >
> > > > > > > > > > Given this situation, we should reconsider our options.
> If
> > the
> > > > > > > > > > PostgreSQL + H2 approach is no longer a viable
> replacement
> > to
> > > > the
> > > > > > > > > > TreeMapMetaStore, I propose an alternative: let's use the
> > NoSQL
> > > > > > > > > > metastore with an InMemory backend.
> > > > > > > > > >
> > > > > > > > > > My reasoning is as follows:
> > > > > > > > > >
> > > > > > > > > > 1. It remains a production-ready metastore, so strictly
> > > > superior to
> > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > >
> > > > > > > > > > 2. The metastore state update logic is the same
> regardless
> > of
> > > > the
> > > > > > > > > > backend, so we're still close to a real production setup
> > (same
> > > > as
> > > > > > with
> > > > > > > > > > an in-memory JDBC driver).
> > > > > > > > > >
> > > > > > > > > > 3. It doesn't require any external dependencies (no
> bundled
> > > > > > driver).
> > > > > > > > > >
> > > > > > > > > > I'd also note that while the NoSQL metastore is already
> > > > included
> > > > > > > > > > today, it is not actionable because no backend is
> > available.
> > > > This
> > > > > > goes
> > > > > > > > > > against the onboarding UX that we're trying to achieve,
> as
> > > > users
> > > > > > > > > > interested in Polaris with the NoSQL metastore cannot
> test
> > this
> > > > > > setup
> > > > > > > > > > with the official image.
> > > > > > > > > >
> > > > > > > > > > Therefore, I think NoSQL + InMemory would serve as a more
> > > > > > appropriate
> > > > > > > > > > default setup for the official Polaris image.
> > > > > > > > > >
> > > > > > > > > > From what I see, it would boil down to adding one line to
> > > > > > > > > > polaris-server or polaris-service:
> > > > > > > > > >
> > > > > > > > > >
> > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > > > > > > > > >
> > > > > > > > > > What are your thoughts on this?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Alex
> > > > > > > > > >
> > > > > > > > > > [1]:
> > > > > > https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra <
> > > > > > adutra@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I wanted to give an update on this topic: I have
> started
> > > > working
> > > > > > on
> > > > > > > > > > > this, but I would like to have [4812] merged first,
> since
> > > > that's
> > > > > > a
> > > > > > > > > > > prerequisite.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Alex
> > > > > > > > > > >
> > > > > > > > > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> > > > > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > I believe the original idea of TreeMapMetastore was
> > that it
> > > > > > looked
> > > > > > > > > very
> > > > > > > > > > > > similar to FoundationDB from an API perspective, so
> it
> > > > served
> > > > > > well
> > > > > > > > > as
> > > > > > > > > > a
> > > > > > > > > > > > test system for the original backend being developed
> > at SF.
> > > > > > > > > > > >
> > > > > > > > > > > > I agree that H2 + JDBC makes sense to me now for the
> > > > project.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra <
> > > > > > adutra@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you for your feedback.
> > > > > > > > > > > > >
> > > > > > > > > > > > > There seems to be a general agreement on the idea
> of
> > > > > > deprecating
> > > > > > > > > > > > > TreeMapMetaStore, coupled with the JDBC + H2
> > solution for
> > > > > > tests.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In most modules, the tests migration won't pose any
> > > > serious
> > > > > > > > > > > > > challenges, as replacing the metastore is generally
> > just
> > > > a
> > > > > > matter
> > > > > > > > > of
> > > > > > > > > > > > > changing the configuration, and making sure the
> > realm is
> > > > > > properly
> > > > > > > > > > > > > bootstrapped.
> > > > > > > > > > > > >
> > > > > > > > > > > > > There will be, however, a few tricky situations in
> > > > > > polaris-core: a
> > > > > > > > > > few
> > > > > > > > > > > > > tests rely on the TreeMapMetaStore, mostly as a
> test
> > > > > > convenience;
> > > > > > > > > but
> > > > > > > > > > > > > there is no obvious replacement for it in that
> > module. I
> > > > am
> > > > > > however
> > > > > > > > > > > > > confident that we can find a solution for that,
> > either
> > > > based
> > > > > > on
> > > > > > > > > > mocks,
> > > > > > > > > > > > > or by bringing in a real metastore.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If no objections are raised, I am going to prepare
> a
> > PR
> > > > for
> > > > > > this.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Alex
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri
> Bourlatchkov <
> > > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for starting this thread.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I also find the TransactionalMetaStoreManagerImpl
> > and
> > > > > > related
> > > > > > > > > call
> > > > > > > > > > paths
> > > > > > > > > > > > > > conuising in Apache Polaris code.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I support promoting H2 to the default (in memory)
> > > > > > Persistence
> > > > > > > > > > backend for
> > > > > > > > > > > > > > getting started cases. This should also resolve
> the
> > > > old H2
> > > > > > > > > > evolution
> > > > > > > > > > > > > thread
> > > > > > > > > > > > > > [1] by ensuring it is used regularly on the same
> > code
> > > > > > paths as
> > > > > > > > > > > > > PostgreSQL.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Deprecating TransactionalMetaStoreManagerImpl for
> > > > removal
> > > > > > also
> > > > > > > > > > sounds
> > > > > > > > > > > > > > reasonable to me. Existing downstream users
> > > > > > > > > > > > > > of TransactionalMetaStoreManagerImpl will have
> > time to
> > > > > > migrate,
> > > > > > > > > or
> > > > > > > > > > even
> > > > > > > > > > > > > > embed that code (per ASF license) into local
> > builds,
> > > > > > during the
> > > > > > > > > > > > > deprecation
> > > > > > > > > > > > > > phase.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]
> > > > > > > > > >
> > > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre Dutra
> <
> > > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I am writing to ask the community whether it is
> > OK to
> > > > > > deprecate
> > > > > > > > > > > > > > > TreeMapMetaStore, as well as all the in-memory
> > > > metastore
> > > > > > > > > manager,
> > > > > > > > > > > > > > > metastore manager factory, and persistence
> types
> > that
> > > > > > rely
> > > > > > > > > > solely on
> > > > > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As we know, these components are test-grade
> only,
> > > > and not
> > > > > > > > > > suitable for
> > > > > > > > > > > > > > > production. They trigger a production readiness
> > > > alert on
> > > > > > > > > Polaris
> > > > > > > > > > > > > > > startup. It's a considerable amount of code
> that
> > is
> > > > > > virtually
> > > > > > > > > > dead in
> > > > > > > > > > > > > > > production.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It's also confusing for developers. E.g. the
> > > > > > "transactional"
> > > > > > > > > > metastore
> > > > > > > > > > > > > > > is not transactional in the JDBC sense of the
> > term,
> > > > and
> > > > > > thus
> > > > > > > > > not
> > > > > > > > > > used
> > > > > > > > > > > > > > > by JDBC persistence. It also has its quirks:
> some
> > > > return
> > > > > > > > > > statuses are
> > > > > > > > > > > > > > > only returned by that manager.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > However, these components are used in tests,
> and
> > I
> > > > agree
> > > > > > that
> > > > > > > > > > it's
> > > > > > > > > > > > > > > useful to have an in-memory version of the
> > > > persistence
> > > > > > layer
> > > > > > > > > for
> > > > > > > > > > > > > > > tests. But we have today two alternatives that
> > are
> > > > imho
> > > > > > > > > superior
> > > > > > > > > > for
> > > > > > > > > > > > > > > tests in polaris-runtime-service:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > - JDBC persistence with H2 backend. There are
> > > > already a
> > > > > > few
> > > > > > > > > tests
> > > > > > > > > > > > > > > using this setup.
> > > > > > > > > > > > > > > - NoSQL persistence with InMemory backend.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Both alternatives test real production-grade
> > > > persistence
> > > > > > code.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > And finally, TreeMapMetaStore is currently the
> > > > default
> > > > > > runtime
> > > > > > > > > > > > > > > persistence in application.properties; and the
> > Helm
> > > > > > chart also
> > > > > > > > > > > > > > > advertises it as the default. These are not
> sane
> > > > > > defaults,
> > > > > > > > > imho.
> > > > > > > > > > It's
> > > > > > > > > > > > > > > always tricky to provide a good default for
> > > > datastores,
> > > > > > but
> > > > > > > > > > since JDBC
> > > > > > > > > > > > > > > persistence is included by default in the
> server
> > > > image,
> > > > > > I think
> > > > > > > > > > that
> > > > > > > > > > > > > > > including the H2 driver by default could give
> us
> > a
> > > > saner
> > > > > > > > > default
> > > > > > > > > > while
> > > > > > > > > > > > > > > keeping the out-of-the-box experience intact
> (the
> > > > > > license is
> > > > > > > > > > Category
> > > > > > > > > > > > > > > A).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Concretely:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On the MetaStoreManagerFactory hierarchy, the
> > > > following
> > > > > > > > > > > > > > > implementations are completely in-memory:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > - InMemoryPolarisMetaStoreManagerFactory: could
> > be
> > > > > > removed
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > -
> InMemoryAtomicOperationMetaStoreManagerFactory:
> > > > could
> > > > > > be
> > > > > > > > > > removed
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > - LocalPolarisMetaStoreManagerFactory: is the
> > base
> > > > class
> > > > > > of the
> > > > > > > > > > two
> > > > > > > > > > > > > > > above; imho it can be removed, but since it's
> an
> > > > abstract
> > > > > > > > > class,
> > > > > > > > > > it
> > > > > > > > > > > > > > > may have been extended outside Polaris. But
> > neither
> > > > JDBC
> > > > > > nor
> > > > > > > > > > NoSQL use
> > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On the PolarisMetaStoreManager hierarchy:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > - TransactionalMetaStoreManagerImpl: is only
> > used by
> > > > > > > > > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC and
> > NoSQL
> > > > do
> > > > > > not use
> > > > > > > > > > it.
> > > > > > > > > > > > > > > Could be removed.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > - TransactionWorkspaceMetaStoreManager however
> > is a
> > > > > > different
> > > > > > > > > > beast,
> > > > > > > > > > > > > > > in spite of the similar name. It is in use
> today
> > on
> > > > the
> > > > > > commit
> > > > > > > > > > path,
> > > > > > > > > > > > > > > so should not be removed.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On the BasePersistence hierarchy:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > - TreeMapTransactionalPersistenceImpl and its
> > > > > > TreeMapMetaStore
> > > > > > > > > > are
> > > > > > > > > > > > > > > only used by
> > InMemoryPolarisMetaStoreManagerFactory,
> > > > and
> > > > > > could
> > > > > > > > > be
> > > > > > > > > > > > > > > removed;
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > - TransactionalPersistence and
> > > > > > > > > AbstractTransactionalPersistence:
> > > > > > > > > > these
> > > > > > > > > > > > > > > are supertypes of
> > TreeMapTransactionalPersistenceImpl
> > > > > > and thus
> > > > > > > > > > only
> > > > > > > > > > > > > > > used for in-memory. They imo can be removed,
> but
> > they
> > > > > > might
> > > > > > > > > have
> > > > > > > > > > been
> > > > > > > > > > > > > > > extended or implemented outside Polaris.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think it's important to keep Polaris code
> tidy
> > by
> > > > > > removing
> > > > > > > > > > unused,
> > > > > > > > > > > > > > > unimplementable, or test-grade only components.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What are your thoughts?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > >
> >
>
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Alex,
Option 1b certainly makes sense to me. I support going in that direction.
I'd propose opening a PR for that merging it before proceeding to test code
changes.
In general, if 1b is acceptable, I'd think H2 should be removed [1] from
tests and replaced with PostgreSQL or NoSQL InMemory. With that in mind,
updating tests on a case-by-case basis seems reasonable to me. We can
decide separately in each PR.
[1] https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
Cheers,
Dmitri.
On Fri, Jul 17, 2026 at 12:59 PM Alexandre Dutra <ad...@apache.org> wrote:
> Hi Dmitri, hi all,
>
> Unfortunately I'm not sure your interpretation matches mine, so let's
> try to wrap up this discussion methodically.
>
> We agreed to deprecate TreeMapMetaStore for removal. But we haven't
> reached consensus on two questions:
>
> 1) What should be the default metastore in the server image?
> 2) How to replace TreeMapMetaStore in tests?
>
> Question 1) is tricky and needs a clear decision. We have only two
> options for a smooth UX (aka, "docker run apache/polaris" just works):
>
> - Option 1a: JDBC+H2
> - Option 1b: NoSQL+InMemory
>
> Option 1a is hard because it requires either my work on [4812]
> (runtime activated datasources) or Yufei's work on [4984] (unmanaged
> datasources). It also *requires* the H2 driver to be bundled by
> default, which some people are not comfortable with. In short, I don't
> see this option being feasible any time soon.
>
> Therefore, I think option 1b is much easier, and that would be my
> preference.
>
> Question 2) is not as urgent, as I'm fairly sure we can find a
> solution as we go. I can see a few options:
> - Option 2a: migrate to JDBC+H2
> - Option 2b: migrate to NoSQL+InMemory
> - Option 2c: decide on a case-by-case basis
>
> My personal preference would be 2c.
>
> Can we try to reach consensus on the two questions please?
>
> Thanks,
> Alex
>
> [4812]: https://github.com/apache/polaris/pull/4812
> [4984]: https://github.com/apache/polaris/pull/4984
>
> On Mon, Jul 6, 2026 at 8:29 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
> >
> > Hi All,
> >
> > My interpretation of the discussion so far is that:
> >
> > 1) We do want to deprecate the TreeMap MetaStore for removal
> >
> > 2) JDBC + PostgreSQL should be used for testing as well as NoSQL +
> MongoDB
> > (I believe it is in fact the case in current CI).
> >
> > 3) Instead of the TreeMap MetaStore, we will use the in-memory NoSQL
> > MetaStore (for lightweight tests)
> >
> > Now, H2 becomes irrevant, IMHO, but removing it is being discussed
> > separately [1].
> >
> > How does that sound?
> >
> > [1] https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> >
> > Thanks,
> > Dmitri.
> >
> > On Fri, Jul 3, 2026 at 5:25 AM Alexandre Dutra <ad...@apache.org>
> wrote:
> >
> > > Hi all,
> > >
> > > I'm confused: security concerns were previously raised here [1] and
> > > here [2] about bundling the H2 driver due to its past record of
> > > critical CVEs.
> > >
> > > As a result, I assumed the JDBC+H2 option had been abandoned. Has this
> > > proposal been brought back into consideration?
> > >
> > > Thanks,
> > > Alex
> > >
> > > [1]: https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
> > > [2]: https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
> > >
> > >
> > > On Fri, Jul 3, 2026 at 8:54 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > > >
> > > > Hi
> > > >
> > > > Yeah fully agree and it was my point when I replied on the thread
> > > > initially.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <fl...@gmail.com> a
> écrit :
> > > >
> > > > > If our goal is to make the default experience closer to production
> > > > > while removing TreeMap, then JDBC with H2 seems like a more natural
> > > > > choice. It exercises the same persistence layer, schema management,
> > > > > and configuration that users will encounter in production, while
> still
> > > > > keeping setup lightweight.
> > > > >
> > > > > Yufei
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <ad...@apache.org>
> > > wrote:
> > > > > >
> > > > > > Hi Russell,
> > > > > >
> > > > > > The existing default metastore shouldn't be viewed merely as a
> > > > > > "relatively simple, self-contained implementation": it has unique
> > > > > > quirks, such as returning result types that no other metastore
> does.
> > > > > >
> > > > > > Choosing the status quo simply to avoid making changes is imho
> the
> > > > > > worst solution.
> > > > > >
> > > > > > Transitioning to the NoSQL metastore as the default remains a net
> > > > > > benefit, even if it is only used by 30% of the user base.
> > > > > >
> > > > > > However, I agree with keeping the test suite on JDBC + H2 by
> default
> > > > > > if that ensures a higher level of confidence in our tests
> ability to
> > > > > > reproduce real production setups.
> > > > > >
> > > > > > Thanks,
> > > > > > Alex
> > > > > >
> > > > > > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
> > > > > > <ru...@gmail.com> wrote:
> > > > > > >
> > > > > > > I agree TreeMap and its associated transactional stack should
> be
> > > > > deprecated.
> > > > > > > It’s test-only, confusing, and not used by JDBC production
> paths.
> > > > > > >
> > > > > > > I’m less convinced that NoSQL + InMemory is a clearly superior
> > > > > replacement
> > > > > > > for TreeMap in tests. At the storage layer it’s essentially the
> > > same
> > > > > class
> > > > > > > of
> > > > > > > thing: an in-memory, non-durable backend. The difference is
> that
> > > it
> > > > > > > exercises
> > > > > > > a completely different MetaStoreManager/persistence stack
> > > > > > > (NoSqlMetaStoreManager
> > > > > > > vs TransactionalMetaStoreManagerImpl), not the JDBC path most
> > > > > deployments
> > > > > > > use. It
> > > > > > > replaces a relatively simple, self-contained implementation
> with
> > > > > something
> > > > > > > that is
> > > > > > > part of a much more complicated persistence stack, while
> remaining
> > > > > > > non-durable
> > > > > > > and test-only at the storage layer.
> > > > > > >
> > > > > > > For the problems originally raised (test-grade defaults, dead
> > > > > production
> > > > > > > path, developer confusion),
> > > > > > > JDBC + H2 seems like an actual fix.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Alex,
> > > > > > > >
> > > > > > > > I tend to agree that the DataSource discussion related to H2
> > > became
> > > > > a bit
> > > > > > > > convoluted, and it's a lot simpler to use the NoSQL in-memory
> > > > > persistence
> > > > > > > > instead.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <
> > > adutra@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > Some updates on this:
> > > > > > > > >
> > > > > > > > > I recently started a separate discussion regarding dynamic
> > > > > datasource
> > > > > > > > > activation at runtime [1]. This was intended as a
> prerequisite
> > > for
> > > > > > > > > adopting H2 as the default JDBC driver, as suggested on
> this
> > > > > thread.
> > > > > > > > >
> > > > > > > > > However, that proposal may end up being rejected.
> > > > > > > > >
> > > > > > > > > Given this situation, we should reconsider our options. If
> the
> > > > > > > > > PostgreSQL + H2 approach is no longer a viable replacement
> to
> > > the
> > > > > > > > > TreeMapMetaStore, I propose an alternative: let's use the
> NoSQL
> > > > > > > > > metastore with an InMemory backend.
> > > > > > > > >
> > > > > > > > > My reasoning is as follows:
> > > > > > > > >
> > > > > > > > > 1. It remains a production-ready metastore, so strictly
> > > superior to
> > > > > > > > > TreeMapMetaStore.
> > > > > > > > >
> > > > > > > > > 2. The metastore state update logic is the same regardless
> of
> > > the
> > > > > > > > > backend, so we're still close to a real production setup
> (same
> > > as
> > > > > with
> > > > > > > > > an in-memory JDBC driver).
> > > > > > > > >
> > > > > > > > > 3. It doesn't require any external dependencies (no bundled
> > > > > driver).
> > > > > > > > >
> > > > > > > > > I'd also note that while the NoSQL metastore is already
> > > included
> > > > > > > > > today, it is not actionable because no backend is
> available.
> > > This
> > > > > goes
> > > > > > > > > against the onboarding UX that we're trying to achieve, as
> > > users
> > > > > > > > > interested in Polaris with the NoSQL metastore cannot test
> this
> > > > > setup
> > > > > > > > > with the official image.
> > > > > > > > >
> > > > > > > > > Therefore, I think NoSQL + InMemory would serve as a more
> > > > > appropriate
> > > > > > > > > default setup for the official Polaris image.
> > > > > > > > >
> > > > > > > > > From what I see, it would boil down to adding one line to
> > > > > > > > > polaris-server or polaris-service:
> > > > > > > > >
> > > > > > > > >
> runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > > > > > > > >
> > > > > > > > > What are your thoughts on this?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Alex
> > > > > > > > >
> > > > > > > > > [1]:
> > > > > https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra <
> > > > > adutra@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I wanted to give an update on this topic: I have started
> > > working
> > > > > on
> > > > > > > > > > this, but I would like to have [4812] merged first, since
> > > that's
> > > > > a
> > > > > > > > > > prerequisite.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Alex
> > > > > > > > > >
> > > > > > > > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> > > > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > I believe the original idea of TreeMapMetastore was
> that it
> > > > > looked
> > > > > > > > very
> > > > > > > > > > > similar to FoundationDB from an API perspective, so it
> > > served
> > > > > well
> > > > > > > > as
> > > > > > > > > a
> > > > > > > > > > > test system for the original backend being developed
> at SF.
> > > > > > > > > > >
> > > > > > > > > > > I agree that H2 + JDBC makes sense to me now for the
> > > project.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra <
> > > > > adutra@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you for your feedback.
> > > > > > > > > > > >
> > > > > > > > > > > > There seems to be a general agreement on the idea of
> > > > > deprecating
> > > > > > > > > > > > TreeMapMetaStore, coupled with the JDBC + H2
> solution for
> > > > > tests.
> > > > > > > > > > > >
> > > > > > > > > > > > In most modules, the tests migration won't pose any
> > > serious
> > > > > > > > > > > > challenges, as replacing the metastore is generally
> just
> > > a
> > > > > matter
> > > > > > > > of
> > > > > > > > > > > > changing the configuration, and making sure the
> realm is
> > > > > properly
> > > > > > > > > > > > bootstrapped.
> > > > > > > > > > > >
> > > > > > > > > > > > There will be, however, a few tricky situations in
> > > > > polaris-core: a
> > > > > > > > > few
> > > > > > > > > > > > tests rely on the TreeMapMetaStore, mostly as a test
> > > > > convenience;
> > > > > > > > but
> > > > > > > > > > > > there is no obvious replacement for it in that
> module. I
> > > am
> > > > > however
> > > > > > > > > > > > confident that we can find a solution for that,
> either
> > > based
> > > > > on
> > > > > > > > > mocks,
> > > > > > > > > > > > or by bringing in a real metastore.
> > > > > > > > > > > >
> > > > > > > > > > > > If no objections are raised, I am going to prepare a
> PR
> > > for
> > > > > this.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Alex
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri Bourlatchkov <
> > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for starting this thread.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I also find the TransactionalMetaStoreManagerImpl
> and
> > > > > related
> > > > > > > > call
> > > > > > > > > paths
> > > > > > > > > > > > > conuising in Apache Polaris code.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I support promoting H2 to the default (in memory)
> > > > > Persistence
> > > > > > > > > backend for
> > > > > > > > > > > > > getting started cases. This should also resolve the
> > > old H2
> > > > > > > > > evolution
> > > > > > > > > > > > thread
> > > > > > > > > > > > > [1] by ensuring it is used regularly on the same
> code
> > > > > paths as
> > > > > > > > > > > > PostgreSQL.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Deprecating TransactionalMetaStoreManagerImpl for
> > > removal
> > > > > also
> > > > > > > > > sounds
> > > > > > > > > > > > > reasonable to me. Existing downstream users
> > > > > > > > > > > > > of TransactionalMetaStoreManagerImpl will have
> time to
> > > > > migrate,
> > > > > > > > or
> > > > > > > > > even
> > > > > > > > > > > > > embed that code (per ASF license) into local
> builds,
> > > > > during the
> > > > > > > > > > > > deprecation
> > > > > > > > > > > > > phase.
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]
> > > > > > > > >
> > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre Dutra <
> > > > > > > > > adutra@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I am writing to ask the community whether it is
> OK to
> > > > > deprecate
> > > > > > > > > > > > > > TreeMapMetaStore, as well as all the in-memory
> > > metastore
> > > > > > > > manager,
> > > > > > > > > > > > > > metastore manager factory, and persistence types
> that
> > > > > rely
> > > > > > > > > solely on
> > > > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As we know, these components are test-grade only,
> > > and not
> > > > > > > > > suitable for
> > > > > > > > > > > > > > production. They trigger a production readiness
> > > alert on
> > > > > > > > Polaris
> > > > > > > > > > > > > > startup. It's a considerable amount of code that
> is
> > > > > virtually
> > > > > > > > > dead in
> > > > > > > > > > > > > > production.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It's also confusing for developers. E.g. the
> > > > > "transactional"
> > > > > > > > > metastore
> > > > > > > > > > > > > > is not transactional in the JDBC sense of the
> term,
> > > and
> > > > > thus
> > > > > > > > not
> > > > > > > > > used
> > > > > > > > > > > > > > by JDBC persistence. It also has its quirks: some
> > > return
> > > > > > > > > statuses are
> > > > > > > > > > > > > > only returned by that manager.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > However, these components are used in tests, and
> I
> > > agree
> > > > > that
> > > > > > > > > it's
> > > > > > > > > > > > > > useful to have an in-memory version of the
> > > persistence
> > > > > layer
> > > > > > > > for
> > > > > > > > > > > > > > tests. But we have today two alternatives that
> are
> > > imho
> > > > > > > > superior
> > > > > > > > > for
> > > > > > > > > > > > > > tests in polaris-runtime-service:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - JDBC persistence with H2 backend. There are
> > > already a
> > > > > few
> > > > > > > > tests
> > > > > > > > > > > > > > using this setup.
> > > > > > > > > > > > > > - NoSQL persistence with InMemory backend.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Both alternatives test real production-grade
> > > persistence
> > > > > code.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > And finally, TreeMapMetaStore is currently the
> > > default
> > > > > runtime
> > > > > > > > > > > > > > persistence in application.properties; and the
> Helm
> > > > > chart also
> > > > > > > > > > > > > > advertises it as the default. These are not sane
> > > > > defaults,
> > > > > > > > imho.
> > > > > > > > > It's
> > > > > > > > > > > > > > always tricky to provide a good default for
> > > datastores,
> > > > > but
> > > > > > > > > since JDBC
> > > > > > > > > > > > > > persistence is included by default in the server
> > > image,
> > > > > I think
> > > > > > > > > that
> > > > > > > > > > > > > > including the H2 driver by default could give us
> a
> > > saner
> > > > > > > > default
> > > > > > > > > while
> > > > > > > > > > > > > > keeping the out-of-the-box experience intact (the
> > > > > license is
> > > > > > > > > Category
> > > > > > > > > > > > > > A).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Concretely:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On the MetaStoreManagerFactory hierarchy, the
> > > following
> > > > > > > > > > > > > > implementations are completely in-memory:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - InMemoryPolarisMetaStoreManagerFactory: could
> be
> > > > > removed
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - InMemoryAtomicOperationMetaStoreManagerFactory:
> > > could
> > > > > be
> > > > > > > > > removed
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - LocalPolarisMetaStoreManagerFactory: is the
> base
> > > class
> > > > > of the
> > > > > > > > > two
> > > > > > > > > > > > > > above; imho it can be removed, but since it's an
> > > abstract
> > > > > > > > class,
> > > > > > > > > it
> > > > > > > > > > > > > > may have been extended outside Polaris. But
> neither
> > > JDBC
> > > > > nor
> > > > > > > > > NoSQL use
> > > > > > > > > > > > > > it.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On the PolarisMetaStoreManager hierarchy:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - TransactionalMetaStoreManagerImpl: is only
> used by
> > > > > > > > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC and
> NoSQL
> > > do
> > > > > not use
> > > > > > > > > it.
> > > > > > > > > > > > > > Could be removed.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - TransactionWorkspaceMetaStoreManager however
> is a
> > > > > different
> > > > > > > > > beast,
> > > > > > > > > > > > > > in spite of the similar name. It is in use today
> on
> > > the
> > > > > commit
> > > > > > > > > path,
> > > > > > > > > > > > > > so should not be removed.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On the BasePersistence hierarchy:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - TreeMapTransactionalPersistenceImpl and its
> > > > > TreeMapMetaStore
> > > > > > > > > are
> > > > > > > > > > > > > > only used by
> InMemoryPolarisMetaStoreManagerFactory,
> > > and
> > > > > could
> > > > > > > > be
> > > > > > > > > > > > > > removed;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - TransactionalPersistence and
> > > > > > > > AbstractTransactionalPersistence:
> > > > > > > > > these
> > > > > > > > > > > > > > are supertypes of
> TreeMapTransactionalPersistenceImpl
> > > > > and thus
> > > > > > > > > only
> > > > > > > > > > > > > > used for in-memory. They imo can be removed, but
> they
> > > > > might
> > > > > > > > have
> > > > > > > > > been
> > > > > > > > > > > > > > extended or implemented outside Polaris.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it's important to keep Polaris code tidy
> by
> > > > > removing
> > > > > > > > > unused,
> > > > > > > > > > > > > > unimplementable, or test-grade only components.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > What are your thoughts?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Alex
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > >
> > >
>
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Posted by Alexandre Dutra <ad...@apache.org>.
Hi Dmitri, hi all,
Unfortunately I'm not sure your interpretation matches mine, so let's
try to wrap up this discussion methodically.
We agreed to deprecate TreeMapMetaStore for removal. But we haven't
reached consensus on two questions:
1) What should be the default metastore in the server image?
2) How to replace TreeMapMetaStore in tests?
Question 1) is tricky and needs a clear decision. We have only two
options for a smooth UX (aka, "docker run apache/polaris" just works):
- Option 1a: JDBC+H2
- Option 1b: NoSQL+InMemory
Option 1a is hard because it requires either my work on [4812]
(runtime activated datasources) or Yufei's work on [4984] (unmanaged
datasources). It also *requires* the H2 driver to be bundled by
default, which some people are not comfortable with. In short, I don't
see this option being feasible any time soon.
Therefore, I think option 1b is much easier, and that would be my preference.
Question 2) is not as urgent, as I'm fairly sure we can find a
solution as we go. I can see a few options:
- Option 2a: migrate to JDBC+H2
- Option 2b: migrate to NoSQL+InMemory
- Option 2c: decide on a case-by-case basis
My personal preference would be 2c.
Can we try to reach consensus on the two questions please?
Thanks,
Alex
[4812]: https://github.com/apache/polaris/pull/4812
[4984]: https://github.com/apache/polaris/pull/4984
On Mon, Jul 6, 2026 at 8:29 PM Dmitri Bourlatchkov <di...@apache.org> wrote:
>
> Hi All,
>
> My interpretation of the discussion so far is that:
>
> 1) We do want to deprecate the TreeMap MetaStore for removal
>
> 2) JDBC + PostgreSQL should be used for testing as well as NoSQL + MongoDB
> (I believe it is in fact the case in current CI).
>
> 3) Instead of the TreeMap MetaStore, we will use the in-memory NoSQL
> MetaStore (for lightweight tests)
>
> Now, H2 becomes irrevant, IMHO, but removing it is being discussed
> separately [1].
>
> How does that sound?
>
> [1] https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
>
> Thanks,
> Dmitri.
>
> On Fri, Jul 3, 2026 at 5:25 AM Alexandre Dutra <ad...@apache.org> wrote:
>
> > Hi all,
> >
> > I'm confused: security concerns were previously raised here [1] and
> > here [2] about bundling the H2 driver due to its past record of
> > critical CVEs.
> >
> > As a result, I assumed the JDBC+H2 option had been abandoned. Has this
> > proposal been brought back into consideration?
> >
> > Thanks,
> > Alex
> >
> > [1]: https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
> > [2]: https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
> >
> >
> > On Fri, Jul 3, 2026 at 8:54 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> > >
> > > Hi
> > >
> > > Yeah fully agree and it was my point when I replied on the thread
> > > initially.
> > >
> > > Regards
> > > JB
> > >
> > > Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <fl...@gmail.com> a écrit :
> > >
> > > > If our goal is to make the default experience closer to production
> > > > while removing TreeMap, then JDBC with H2 seems like a more natural
> > > > choice. It exercises the same persistence layer, schema management,
> > > > and configuration that users will encounter in production, while still
> > > > keeping setup lightweight.
> > > >
> > > > Yufei
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <ad...@apache.org>
> > wrote:
> > > > >
> > > > > Hi Russell,
> > > > >
> > > > > The existing default metastore shouldn't be viewed merely as a
> > > > > "relatively simple, self-contained implementation": it has unique
> > > > > quirks, such as returning result types that no other metastore does.
> > > > >
> > > > > Choosing the status quo simply to avoid making changes is imho the
> > > > > worst solution.
> > > > >
> > > > > Transitioning to the NoSQL metastore as the default remains a net
> > > > > benefit, even if it is only used by 30% of the user base.
> > > > >
> > > > > However, I agree with keeping the test suite on JDBC + H2 by default
> > > > > if that ensures a higher level of confidence in our tests ability to
> > > > > reproduce real production setups.
> > > > >
> > > > > Thanks,
> > > > > Alex
> > > > >
> > > > > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
> > > > > <ru...@gmail.com> wrote:
> > > > > >
> > > > > > I agree TreeMap and its associated transactional stack should be
> > > > deprecated.
> > > > > > It’s test-only, confusing, and not used by JDBC production paths.
> > > > > >
> > > > > > I’m less convinced that NoSQL + InMemory is a clearly superior
> > > > replacement
> > > > > > for TreeMap in tests. At the storage layer it’s essentially the
> > same
> > > > class
> > > > > > of
> > > > > > thing: an in-memory, non-durable backend. The difference is that
> > it
> > > > > > exercises
> > > > > > a completely different MetaStoreManager/persistence stack
> > > > > > (NoSqlMetaStoreManager
> > > > > > vs TransactionalMetaStoreManagerImpl), not the JDBC path most
> > > > deployments
> > > > > > use. It
> > > > > > replaces a relatively simple, self-contained implementation with
> > > > something
> > > > > > that is
> > > > > > part of a much more complicated persistence stack, while remaining
> > > > > > non-durable
> > > > > > and test-only at the storage layer.
> > > > > >
> > > > > > For the problems originally raised (test-grade defaults, dead
> > > > production
> > > > > > path, developer confusion),
> > > > > > JDBC + H2 seems like an actual fix.
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Alex,
> > > > > > >
> > > > > > > I tend to agree that the DataSource discussion related to H2
> > became
> > > > a bit
> > > > > > > convoluted, and it's a lot simpler to use the NoSQL in-memory
> > > > persistence
> > > > > > > instead.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <
> > adutra@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Some updates on this:
> > > > > > > >
> > > > > > > > I recently started a separate discussion regarding dynamic
> > > > datasource
> > > > > > > > activation at runtime [1]. This was intended as a prerequisite
> > for
> > > > > > > > adopting H2 as the default JDBC driver, as suggested on this
> > > > thread.
> > > > > > > >
> > > > > > > > However, that proposal may end up being rejected.
> > > > > > > >
> > > > > > > > Given this situation, we should reconsider our options. If the
> > > > > > > > PostgreSQL + H2 approach is no longer a viable replacement to
> > the
> > > > > > > > TreeMapMetaStore, I propose an alternative: let's use the NoSQL
> > > > > > > > metastore with an InMemory backend.
> > > > > > > >
> > > > > > > > My reasoning is as follows:
> > > > > > > >
> > > > > > > > 1. It remains a production-ready metastore, so strictly
> > superior to
> > > > > > > > TreeMapMetaStore.
> > > > > > > >
> > > > > > > > 2. The metastore state update logic is the same regardless of
> > the
> > > > > > > > backend, so we're still close to a real production setup (same
> > as
> > > > with
> > > > > > > > an in-memory JDBC driver).
> > > > > > > >
> > > > > > > > 3. It doesn't require any external dependencies (no bundled
> > > > driver).
> > > > > > > >
> > > > > > > > I'd also note that while the NoSQL metastore is already
> > included
> > > > > > > > today, it is not actionable because no backend is available.
> > This
> > > > goes
> > > > > > > > against the onboarding UX that we're trying to achieve, as
> > users
> > > > > > > > interested in Polaris with the NoSQL metastore cannot test this
> > > > setup
> > > > > > > > with the official image.
> > > > > > > >
> > > > > > > > Therefore, I think NoSQL + InMemory would serve as a more
> > > > appropriate
> > > > > > > > default setup for the official Polaris image.
> > > > > > > >
> > > > > > > > From what I see, it would boil down to adding one line to
> > > > > > > > polaris-server or polaris-service:
> > > > > > > >
> > > > > > > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > > > > > > >
> > > > > > > > What are your thoughts on this?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Alex
> > > > > > > >
> > > > > > > > [1]:
> > > > https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra <
> > > > adutra@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I wanted to give an update on this topic: I have started
> > working
> > > > on
> > > > > > > > > this, but I would like to have [4812] merged first, since
> > that's
> > > > a
> > > > > > > > > prerequisite.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Alex
> > > > > > > > >
> > > > > > > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > > > > > > >
> > > > > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> > > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > I believe the original idea of TreeMapMetastore was that it
> > > > looked
> > > > > > > very
> > > > > > > > > > similar to FoundationDB from an API perspective, so it
> > served
> > > > well
> > > > > > > as
> > > > > > > > a
> > > > > > > > > > test system for the original backend being developed at SF.
> > > > > > > > > >
> > > > > > > > > > I agree that H2 + JDBC makes sense to me now for the
> > project.
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra <
> > > > adutra@apache.org>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > Thank you for your feedback.
> > > > > > > > > > >
> > > > > > > > > > > There seems to be a general agreement on the idea of
> > > > deprecating
> > > > > > > > > > > TreeMapMetaStore, coupled with the JDBC + H2 solution for
> > > > tests.
> > > > > > > > > > >
> > > > > > > > > > > In most modules, the tests migration won't pose any
> > serious
> > > > > > > > > > > challenges, as replacing the metastore is generally just
> > a
> > > > matter
> > > > > > > of
> > > > > > > > > > > changing the configuration, and making sure the realm is
> > > > properly
> > > > > > > > > > > bootstrapped.
> > > > > > > > > > >
> > > > > > > > > > > There will be, however, a few tricky situations in
> > > > polaris-core: a
> > > > > > > > few
> > > > > > > > > > > tests rely on the TreeMapMetaStore, mostly as a test
> > > > convenience;
> > > > > > > but
> > > > > > > > > > > there is no obvious replacement for it in that module. I
> > am
> > > > however
> > > > > > > > > > > confident that we can find a solution for that, either
> > based
> > > > on
> > > > > > > > mocks,
> > > > > > > > > > > or by bringing in a real metastore.
> > > > > > > > > > >
> > > > > > > > > > > If no objections are raised, I am going to prepare a PR
> > for
> > > > this.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Alex
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri Bourlatchkov <
> > > > > > > > dimas@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for starting this thread.
> > > > > > > > > > > >
> > > > > > > > > > > > I also find the TransactionalMetaStoreManagerImpl and
> > > > related
> > > > > > > call
> > > > > > > > paths
> > > > > > > > > > > > conuising in Apache Polaris code.
> > > > > > > > > > > >
> > > > > > > > > > > > I support promoting H2 to the default (in memory)
> > > > Persistence
> > > > > > > > backend for
> > > > > > > > > > > > getting started cases. This should also resolve the
> > old H2
> > > > > > > > evolution
> > > > > > > > > > > thread
> > > > > > > > > > > > [1] by ensuring it is used regularly on the same code
> > > > paths as
> > > > > > > > > > > PostgreSQL.
> > > > > > > > > > > >
> > > > > > > > > > > > Deprecating TransactionalMetaStoreManagerImpl for
> > removal
> > > > also
> > > > > > > > sounds
> > > > > > > > > > > > reasonable to me. Existing downstream users
> > > > > > > > > > > > of TransactionalMetaStoreManagerImpl will have time to
> > > > migrate,
> > > > > > > or
> > > > > > > > even
> > > > > > > > > > > > embed that code (per ASF license) into local builds,
> > > > during the
> > > > > > > > > > > deprecation
> > > > > > > > > > > > phase.
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> > > > > > > >
> > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Dmitri.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre Dutra <
> > > > > > > > adutra@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I am writing to ask the community whether it is OK to
> > > > deprecate
> > > > > > > > > > > > > TreeMapMetaStore, as well as all the in-memory
> > metastore
> > > > > > > manager,
> > > > > > > > > > > > > metastore manager factory, and persistence types that
> > > > rely
> > > > > > > > solely on
> > > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > > >
> > > > > > > > > > > > > As we know, these components are test-grade only,
> > and not
> > > > > > > > suitable for
> > > > > > > > > > > > > production. They trigger a production readiness
> > alert on
> > > > > > > Polaris
> > > > > > > > > > > > > startup. It's a considerable amount of code that is
> > > > virtually
> > > > > > > > dead in
> > > > > > > > > > > > > production.
> > > > > > > > > > > > >
> > > > > > > > > > > > > It's also confusing for developers. E.g. the
> > > > "transactional"
> > > > > > > > metastore
> > > > > > > > > > > > > is not transactional in the JDBC sense of the term,
> > and
> > > > thus
> > > > > > > not
> > > > > > > > used
> > > > > > > > > > > > > by JDBC persistence. It also has its quirks: some
> > return
> > > > > > > > statuses are
> > > > > > > > > > > > > only returned by that manager.
> > > > > > > > > > > > >
> > > > > > > > > > > > > However, these components are used in tests, and I
> > agree
> > > > that
> > > > > > > > it's
> > > > > > > > > > > > > useful to have an in-memory version of the
> > persistence
> > > > layer
> > > > > > > for
> > > > > > > > > > > > > tests. But we have today two alternatives that are
> > imho
> > > > > > > superior
> > > > > > > > for
> > > > > > > > > > > > > tests in polaris-runtime-service:
> > > > > > > > > > > > >
> > > > > > > > > > > > > - JDBC persistence with H2 backend. There are
> > already a
> > > > few
> > > > > > > tests
> > > > > > > > > > > > > using this setup.
> > > > > > > > > > > > > - NoSQL persistence with InMemory backend.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Both alternatives test real production-grade
> > persistence
> > > > code.
> > > > > > > > > > > > >
> > > > > > > > > > > > > And finally, TreeMapMetaStore is currently the
> > default
> > > > runtime
> > > > > > > > > > > > > persistence in application.properties; and the Helm
> > > > chart also
> > > > > > > > > > > > > advertises it as the default. These are not sane
> > > > defaults,
> > > > > > > imho.
> > > > > > > > It's
> > > > > > > > > > > > > always tricky to provide a good default for
> > datastores,
> > > > but
> > > > > > > > since JDBC
> > > > > > > > > > > > > persistence is included by default in the server
> > image,
> > > > I think
> > > > > > > > that
> > > > > > > > > > > > > including the H2 driver by default could give us a
> > saner
> > > > > > > default
> > > > > > > > while
> > > > > > > > > > > > > keeping the out-of-the-box experience intact (the
> > > > license is
> > > > > > > > Category
> > > > > > > > > > > > > A).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Concretely:
> > > > > > > > > > > > >
> > > > > > > > > > > > > On the MetaStoreManagerFactory hierarchy, the
> > following
> > > > > > > > > > > > > implementations are completely in-memory:
> > > > > > > > > > > > >
> > > > > > > > > > > > > - InMemoryPolarisMetaStoreManagerFactory: could be
> > > > removed
> > > > > > > > > > > > >
> > > > > > > > > > > > > - InMemoryAtomicOperationMetaStoreManagerFactory:
> > could
> > > > be
> > > > > > > > removed
> > > > > > > > > > > > >
> > > > > > > > > > > > > - LocalPolarisMetaStoreManagerFactory: is the base
> > class
> > > > of the
> > > > > > > > two
> > > > > > > > > > > > > above; imho it can be removed, but since it's an
> > abstract
> > > > > > > class,
> > > > > > > > it
> > > > > > > > > > > > > may have been extended outside Polaris. But neither
> > JDBC
> > > > nor
> > > > > > > > NoSQL use
> > > > > > > > > > > > > it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On the PolarisMetaStoreManager hierarchy:
> > > > > > > > > > > > >
> > > > > > > > > > > > > - TransactionalMetaStoreManagerImpl: is only used by
> > > > > > > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC and NoSQL
> > do
> > > > not use
> > > > > > > > it.
> > > > > > > > > > > > > Could be removed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > - TransactionWorkspaceMetaStoreManager however is a
> > > > different
> > > > > > > > beast,
> > > > > > > > > > > > > in spite of the similar name. It is in use today on
> > the
> > > > commit
> > > > > > > > path,
> > > > > > > > > > > > > so should not be removed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On the BasePersistence hierarchy:
> > > > > > > > > > > > >
> > > > > > > > > > > > > - TreeMapTransactionalPersistenceImpl and its
> > > > TreeMapMetaStore
> > > > > > > > are
> > > > > > > > > > > > > only used by InMemoryPolarisMetaStoreManagerFactory,
> > and
> > > > could
> > > > > > > be
> > > > > > > > > > > > > removed;
> > > > > > > > > > > > >
> > > > > > > > > > > > > - TransactionalPersistence and
> > > > > > > AbstractTransactionalPersistence:
> > > > > > > > these
> > > > > > > > > > > > > are supertypes of TreeMapTransactionalPersistenceImpl
> > > > and thus
> > > > > > > > only
> > > > > > > > > > > > > used for in-memory. They imo can be removed, but they
> > > > might
> > > > > > > have
> > > > > > > > been
> > > > > > > > > > > > > extended or implemented outside Polaris.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think it's important to keep Polaris code tidy by
> > > > removing
> > > > > > > > unused,
> > > > > > > > > > > > > unimplementable, or test-grade only components.
> > > > > > > > > > > > >
> > > > > > > > > > > > > What are your thoughts?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Alex
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > >
> >
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi All,
My interpretation of the discussion so far is that:
1) We do want to deprecate the TreeMap MetaStore for removal
2) JDBC + PostgreSQL should be used for testing as well as NoSQL + MongoDB
(I believe it is in fact the case in current CI).
3) Instead of the TreeMap MetaStore, we will use the in-memory NoSQL
MetaStore (for lightweight tests)
Now, H2 becomes irrevant, IMHO, but removing it is being discussed
separately [1].
How does that sound?
[1] https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
Thanks,
Dmitri.
On Fri, Jul 3, 2026 at 5:25 AM Alexandre Dutra <ad...@apache.org> wrote:
> Hi all,
>
> I'm confused: security concerns were previously raised here [1] and
> here [2] about bundling the H2 driver due to its past record of
> critical CVEs.
>
> As a result, I assumed the JDBC+H2 option had been abandoned. Has this
> proposal been brought back into consideration?
>
> Thanks,
> Alex
>
> [1]: https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
> [2]: https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
>
>
> On Fri, Jul 3, 2026 at 8:54 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >
> > Hi
> >
> > Yeah fully agree and it was my point when I replied on the thread
> > initially.
> >
> > Regards
> > JB
> >
> > Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <fl...@gmail.com> a écrit :
> >
> > > If our goal is to make the default experience closer to production
> > > while removing TreeMap, then JDBC with H2 seems like a more natural
> > > choice. It exercises the same persistence layer, schema management,
> > > and configuration that users will encounter in production, while still
> > > keeping setup lightweight.
> > >
> > > Yufei
> > >
> > > Yufei
> > >
> > >
> > > On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <ad...@apache.org>
> wrote:
> > > >
> > > > Hi Russell,
> > > >
> > > > The existing default metastore shouldn't be viewed merely as a
> > > > "relatively simple, self-contained implementation": it has unique
> > > > quirks, such as returning result types that no other metastore does.
> > > >
> > > > Choosing the status quo simply to avoid making changes is imho the
> > > > worst solution.
> > > >
> > > > Transitioning to the NoSQL metastore as the default remains a net
> > > > benefit, even if it is only used by 30% of the user base.
> > > >
> > > > However, I agree with keeping the test suite on JDBC + H2 by default
> > > > if that ensures a higher level of confidence in our tests ability to
> > > > reproduce real production setups.
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > > > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
> > > > <ru...@gmail.com> wrote:
> > > > >
> > > > > I agree TreeMap and its associated transactional stack should be
> > > deprecated.
> > > > > It’s test-only, confusing, and not used by JDBC production paths.
> > > > >
> > > > > I’m less convinced that NoSQL + InMemory is a clearly superior
> > > replacement
> > > > > for TreeMap in tests. At the storage layer it’s essentially the
> same
> > > class
> > > > > of
> > > > > thing: an in-memory, non-durable backend. The difference is that
> it
> > > > > exercises
> > > > > a completely different MetaStoreManager/persistence stack
> > > > > (NoSqlMetaStoreManager
> > > > > vs TransactionalMetaStoreManagerImpl), not the JDBC path most
> > > deployments
> > > > > use. It
> > > > > replaces a relatively simple, self-contained implementation with
> > > something
> > > > > that is
> > > > > part of a much more complicated persistence stack, while remaining
> > > > > non-durable
> > > > > and test-only at the storage layer.
> > > > >
> > > > > For the problems originally raised (test-grade defaults, dead
> > > production
> > > > > path, developer confusion),
> > > > > JDBC + H2 seems like an actual fix.
> > > > >
> > > > >
> > > > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Alex,
> > > > > >
> > > > > > I tend to agree that the DataSource discussion related to H2
> became
> > > a bit
> > > > > > convoluted, and it's a lot simpler to use the NoSQL in-memory
> > > persistence
> > > > > > instead.
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <
> adutra@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Some updates on this:
> > > > > > >
> > > > > > > I recently started a separate discussion regarding dynamic
> > > datasource
> > > > > > > activation at runtime [1]. This was intended as a prerequisite
> for
> > > > > > > adopting H2 as the default JDBC driver, as suggested on this
> > > thread.
> > > > > > >
> > > > > > > However, that proposal may end up being rejected.
> > > > > > >
> > > > > > > Given this situation, we should reconsider our options. If the
> > > > > > > PostgreSQL + H2 approach is no longer a viable replacement to
> the
> > > > > > > TreeMapMetaStore, I propose an alternative: let's use the NoSQL
> > > > > > > metastore with an InMemory backend.
> > > > > > >
> > > > > > > My reasoning is as follows:
> > > > > > >
> > > > > > > 1. It remains a production-ready metastore, so strictly
> superior to
> > > > > > > TreeMapMetaStore.
> > > > > > >
> > > > > > > 2. The metastore state update logic is the same regardless of
> the
> > > > > > > backend, so we're still close to a real production setup (same
> as
> > > with
> > > > > > > an in-memory JDBC driver).
> > > > > > >
> > > > > > > 3. It doesn't require any external dependencies (no bundled
> > > driver).
> > > > > > >
> > > > > > > I'd also note that while the NoSQL metastore is already
> included
> > > > > > > today, it is not actionable because no backend is available.
> This
> > > goes
> > > > > > > against the onboarding UX that we're trying to achieve, as
> users
> > > > > > > interested in Polaris with the NoSQL metastore cannot test this
> > > setup
> > > > > > > with the official image.
> > > > > > >
> > > > > > > Therefore, I think NoSQL + InMemory would serve as a more
> > > appropriate
> > > > > > > default setup for the official Polaris image.
> > > > > > >
> > > > > > > From what I see, it would boil down to adding one line to
> > > > > > > polaris-server or polaris-service:
> > > > > > >
> > > > > > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > > > > > >
> > > > > > > What are your thoughts on this?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Alex
> > > > > > >
> > > > > > > [1]:
> > > https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra <
> > > adutra@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I wanted to give an update on this topic: I have started
> working
> > > on
> > > > > > > > this, but I would like to have [4812] merged first, since
> that's
> > > a
> > > > > > > > prerequisite.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Alex
> > > > > > > >
> > > > > > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > > > > > >
> > > > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> > > > > > > > <ru...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > I believe the original idea of TreeMapMetastore was that it
> > > looked
> > > > > > very
> > > > > > > > > similar to FoundationDB from an API perspective, so it
> served
> > > well
> > > > > > as
> > > > > > > a
> > > > > > > > > test system for the original backend being developed at SF.
> > > > > > > > >
> > > > > > > > > I agree that H2 + JDBC makes sense to me now for the
> project.
> > > > > > > > >
> > > > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra <
> > > adutra@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > Thank you for your feedback.
> > > > > > > > > >
> > > > > > > > > > There seems to be a general agreement on the idea of
> > > deprecating
> > > > > > > > > > TreeMapMetaStore, coupled with the JDBC + H2 solution for
> > > tests.
> > > > > > > > > >
> > > > > > > > > > In most modules, the tests migration won't pose any
> serious
> > > > > > > > > > challenges, as replacing the metastore is generally just
> a
> > > matter
> > > > > > of
> > > > > > > > > > changing the configuration, and making sure the realm is
> > > properly
> > > > > > > > > > bootstrapped.
> > > > > > > > > >
> > > > > > > > > > There will be, however, a few tricky situations in
> > > polaris-core: a
> > > > > > > few
> > > > > > > > > > tests rely on the TreeMapMetaStore, mostly as a test
> > > convenience;
> > > > > > but
> > > > > > > > > > there is no obvious replacement for it in that module. I
> am
> > > however
> > > > > > > > > > confident that we can find a solution for that, either
> based
> > > on
> > > > > > > mocks,
> > > > > > > > > > or by bringing in a real metastore.
> > > > > > > > > >
> > > > > > > > > > If no objections are raised, I am going to prepare a PR
> for
> > > this.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Alex
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri Bourlatchkov <
> > > > > > > dimas@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Alex,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for starting this thread.
> > > > > > > > > > >
> > > > > > > > > > > I also find the TransactionalMetaStoreManagerImpl and
> > > related
> > > > > > call
> > > > > > > paths
> > > > > > > > > > > conuising in Apache Polaris code.
> > > > > > > > > > >
> > > > > > > > > > > I support promoting H2 to the default (in memory)
> > > Persistence
> > > > > > > backend for
> > > > > > > > > > > getting started cases. This should also resolve the
> old H2
> > > > > > > evolution
> > > > > > > > > > thread
> > > > > > > > > > > [1] by ensuring it is used regularly on the same code
> > > paths as
> > > > > > > > > > PostgreSQL.
> > > > > > > > > > >
> > > > > > > > > > > Deprecating TransactionalMetaStoreManagerImpl for
> removal
> > > also
> > > > > > > sounds
> > > > > > > > > > > reasonable to me. Existing downstream users
> > > > > > > > > > > of TransactionalMetaStoreManagerImpl will have time to
> > > migrate,
> > > > > > or
> > > > > > > even
> > > > > > > > > > > embed that code (per ASF license) into local builds,
> > > during the
> > > > > > > > > > deprecation
> > > > > > > > > > > phase.
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > >
> https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre Dutra <
> > > > > > > adutra@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I am writing to ask the community whether it is OK to
> > > deprecate
> > > > > > > > > > > > TreeMapMetaStore, as well as all the in-memory
> metastore
> > > > > > manager,
> > > > > > > > > > > > metastore manager factory, and persistence types that
> > > rely
> > > > > > > solely on
> > > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > > >
> > > > > > > > > > > > As we know, these components are test-grade only,
> and not
> > > > > > > suitable for
> > > > > > > > > > > > production. They trigger a production readiness
> alert on
> > > > > > Polaris
> > > > > > > > > > > > startup. It's a considerable amount of code that is
> > > virtually
> > > > > > > dead in
> > > > > > > > > > > > production.
> > > > > > > > > > > >
> > > > > > > > > > > > It's also confusing for developers. E.g. the
> > > "transactional"
> > > > > > > metastore
> > > > > > > > > > > > is not transactional in the JDBC sense of the term,
> and
> > > thus
> > > > > > not
> > > > > > > used
> > > > > > > > > > > > by JDBC persistence. It also has its quirks: some
> return
> > > > > > > statuses are
> > > > > > > > > > > > only returned by that manager.
> > > > > > > > > > > >
> > > > > > > > > > > > However, these components are used in tests, and I
> agree
> > > that
> > > > > > > it's
> > > > > > > > > > > > useful to have an in-memory version of the
> persistence
> > > layer
> > > > > > for
> > > > > > > > > > > > tests. But we have today two alternatives that are
> imho
> > > > > > superior
> > > > > > > for
> > > > > > > > > > > > tests in polaris-runtime-service:
> > > > > > > > > > > >
> > > > > > > > > > > > - JDBC persistence with H2 backend. There are
> already a
> > > few
> > > > > > tests
> > > > > > > > > > > > using this setup.
> > > > > > > > > > > > - NoSQL persistence with InMemory backend.
> > > > > > > > > > > >
> > > > > > > > > > > > Both alternatives test real production-grade
> persistence
> > > code.
> > > > > > > > > > > >
> > > > > > > > > > > > And finally, TreeMapMetaStore is currently the
> default
> > > runtime
> > > > > > > > > > > > persistence in application.properties; and the Helm
> > > chart also
> > > > > > > > > > > > advertises it as the default. These are not sane
> > > defaults,
> > > > > > imho.
> > > > > > > It's
> > > > > > > > > > > > always tricky to provide a good default for
> datastores,
> > > but
> > > > > > > since JDBC
> > > > > > > > > > > > persistence is included by default in the server
> image,
> > > I think
> > > > > > > that
> > > > > > > > > > > > including the H2 driver by default could give us a
> saner
> > > > > > default
> > > > > > > while
> > > > > > > > > > > > keeping the out-of-the-box experience intact (the
> > > license is
> > > > > > > Category
> > > > > > > > > > > > A).
> > > > > > > > > > > >
> > > > > > > > > > > > Concretely:
> > > > > > > > > > > >
> > > > > > > > > > > > On the MetaStoreManagerFactory hierarchy, the
> following
> > > > > > > > > > > > implementations are completely in-memory:
> > > > > > > > > > > >
> > > > > > > > > > > > - InMemoryPolarisMetaStoreManagerFactory: could be
> > > removed
> > > > > > > > > > > >
> > > > > > > > > > > > - InMemoryAtomicOperationMetaStoreManagerFactory:
> could
> > > be
> > > > > > > removed
> > > > > > > > > > > >
> > > > > > > > > > > > - LocalPolarisMetaStoreManagerFactory: is the base
> class
> > > of the
> > > > > > > two
> > > > > > > > > > > > above; imho it can be removed, but since it's an
> abstract
> > > > > > class,
> > > > > > > it
> > > > > > > > > > > > may have been extended outside Polaris. But neither
> JDBC
> > > nor
> > > > > > > NoSQL use
> > > > > > > > > > > > it.
> > > > > > > > > > > >
> > > > > > > > > > > > On the PolarisMetaStoreManager hierarchy:
> > > > > > > > > > > >
> > > > > > > > > > > > - TransactionalMetaStoreManagerImpl: is only used by
> > > > > > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC and NoSQL
> do
> > > not use
> > > > > > > it.
> > > > > > > > > > > > Could be removed.
> > > > > > > > > > > >
> > > > > > > > > > > > - TransactionWorkspaceMetaStoreManager however is a
> > > different
> > > > > > > beast,
> > > > > > > > > > > > in spite of the similar name. It is in use today on
> the
> > > commit
> > > > > > > path,
> > > > > > > > > > > > so should not be removed.
> > > > > > > > > > > >
> > > > > > > > > > > > On the BasePersistence hierarchy:
> > > > > > > > > > > >
> > > > > > > > > > > > - TreeMapTransactionalPersistenceImpl and its
> > > TreeMapMetaStore
> > > > > > > are
> > > > > > > > > > > > only used by InMemoryPolarisMetaStoreManagerFactory,
> and
> > > could
> > > > > > be
> > > > > > > > > > > > removed;
> > > > > > > > > > > >
> > > > > > > > > > > > - TransactionalPersistence and
> > > > > > AbstractTransactionalPersistence:
> > > > > > > these
> > > > > > > > > > > > are supertypes of TreeMapTransactionalPersistenceImpl
> > > and thus
> > > > > > > only
> > > > > > > > > > > > used for in-memory. They imo can be removed, but they
> > > might
> > > > > > have
> > > > > > > been
> > > > > > > > > > > > extended or implemented outside Polaris.
> > > > > > > > > > > >
> > > > > > > > > > > > I think it's important to keep Polaris code tidy by
> > > removing
> > > > > > > unused,
> > > > > > > > > > > > unimplementable, or test-grade only components.
> > > > > > > > > > > >
> > > > > > > > > > > > What are your thoughts?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Alex
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > > >
> > >
>
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Posted by Alexandre Dutra <ad...@apache.org>.
Hi all,
I'm confused: security concerns were previously raised here [1] and
here [2] about bundling the H2 driver due to its past record of
critical CVEs.
As a result, I assumed the JDBC+H2 option had been abandoned. Has this
proposal been brought back into consideration?
Thanks,
Alex
[1]: https://lists.apache.org/thread/jq41dpqoys551wf30qkf0wcwxssxo3qj
[2]: https://lists.apache.org/thread/qdzb3lt8fwbpkjyd3lh07ylxr6wpwn1r
On Fri, Jul 3, 2026 at 8:54 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi
>
> Yeah fully agree and it was my point when I replied on the thread
> initially.
>
> Regards
> JB
>
> Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <fl...@gmail.com> a écrit :
>
> > If our goal is to make the default experience closer to production
> > while removing TreeMap, then JDBC with H2 seems like a more natural
> > choice. It exercises the same persistence layer, schema management,
> > and configuration that users will encounter in production, while still
> > keeping setup lightweight.
> >
> > Yufei
> >
> > Yufei
> >
> >
> > On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <ad...@apache.org> wrote:
> > >
> > > Hi Russell,
> > >
> > > The existing default metastore shouldn't be viewed merely as a
> > > "relatively simple, self-contained implementation": it has unique
> > > quirks, such as returning result types that no other metastore does.
> > >
> > > Choosing the status quo simply to avoid making changes is imho the
> > > worst solution.
> > >
> > > Transitioning to the NoSQL metastore as the default remains a net
> > > benefit, even if it is only used by 30% of the user base.
> > >
> > > However, I agree with keeping the test suite on JDBC + H2 by default
> > > if that ensures a higher level of confidence in our tests ability to
> > > reproduce real production setups.
> > >
> > > Thanks,
> > > Alex
> > >
> > > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
> > > <ru...@gmail.com> wrote:
> > > >
> > > > I agree TreeMap and its associated transactional stack should be
> > deprecated.
> > > > It’s test-only, confusing, and not used by JDBC production paths.
> > > >
> > > > I’m less convinced that NoSQL + InMemory is a clearly superior
> > replacement
> > > > for TreeMap in tests. At the storage layer it’s essentially the same
> > class
> > > > of
> > > > thing: an in-memory, non-durable backend. The difference is that it
> > > > exercises
> > > > a completely different MetaStoreManager/persistence stack
> > > > (NoSqlMetaStoreManager
> > > > vs TransactionalMetaStoreManagerImpl), not the JDBC path most
> > deployments
> > > > use. It
> > > > replaces a relatively simple, self-contained implementation with
> > something
> > > > that is
> > > > part of a much more complicated persistence stack, while remaining
> > > > non-durable
> > > > and test-only at the storage layer.
> > > >
> > > > For the problems originally raised (test-grade defaults, dead
> > production
> > > > path, developer confusion),
> > > > JDBC + H2 seems like an actual fix.
> > > >
> > > >
> > > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <di...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Alex,
> > > > >
> > > > > I tend to agree that the DataSource discussion related to H2 became
> > a bit
> > > > > convoluted, and it's a lot simpler to use the NoSQL in-memory
> > persistence
> > > > > instead.
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <ad...@apache.org>
> > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Some updates on this:
> > > > > >
> > > > > > I recently started a separate discussion regarding dynamic
> > datasource
> > > > > > activation at runtime [1]. This was intended as a prerequisite for
> > > > > > adopting H2 as the default JDBC driver, as suggested on this
> > thread.
> > > > > >
> > > > > > However, that proposal may end up being rejected.
> > > > > >
> > > > > > Given this situation, we should reconsider our options. If the
> > > > > > PostgreSQL + H2 approach is no longer a viable replacement to the
> > > > > > TreeMapMetaStore, I propose an alternative: let's use the NoSQL
> > > > > > metastore with an InMemory backend.
> > > > > >
> > > > > > My reasoning is as follows:
> > > > > >
> > > > > > 1. It remains a production-ready metastore, so strictly superior to
> > > > > > TreeMapMetaStore.
> > > > > >
> > > > > > 2. The metastore state update logic is the same regardless of the
> > > > > > backend, so we're still close to a real production setup (same as
> > with
> > > > > > an in-memory JDBC driver).
> > > > > >
> > > > > > 3. It doesn't require any external dependencies (no bundled
> > driver).
> > > > > >
> > > > > > I'd also note that while the NoSQL metastore is already included
> > > > > > today, it is not actionable because no backend is available. This
> > goes
> > > > > > against the onboarding UX that we're trying to achieve, as users
> > > > > > interested in Polaris with the NoSQL metastore cannot test this
> > setup
> > > > > > with the official image.
> > > > > >
> > > > > > Therefore, I think NoSQL + InMemory would serve as a more
> > appropriate
> > > > > > default setup for the official Polaris image.
> > > > > >
> > > > > > From what I see, it would boil down to adding one line to
> > > > > > polaris-server or polaris-service:
> > > > > >
> > > > > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > > > > >
> > > > > > What are your thoughts on this?
> > > > > >
> > > > > > Thanks,
> > > > > > Alex
> > > > > >
> > > > > > [1]:
> > https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra <
> > adutra@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I wanted to give an update on this topic: I have started working
> > on
> > > > > > > this, but I would like to have [4812] merged first, since that's
> > a
> > > > > > > prerequisite.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Alex
> > > > > > >
> > > > > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > > > > >
> > > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> > > > > > > <ru...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > I believe the original idea of TreeMapMetastore was that it
> > looked
> > > > > very
> > > > > > > > similar to FoundationDB from an API perspective, so it served
> > well
> > > > > as
> > > > > > a
> > > > > > > > test system for the original backend being developed at SF.
> > > > > > > >
> > > > > > > > I agree that H2 + JDBC makes sense to me now for the project.
> > > > > > > >
> > > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra <
> > adutra@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > Thank you for your feedback.
> > > > > > > > >
> > > > > > > > > There seems to be a general agreement on the idea of
> > deprecating
> > > > > > > > > TreeMapMetaStore, coupled with the JDBC + H2 solution for
> > tests.
> > > > > > > > >
> > > > > > > > > In most modules, the tests migration won't pose any serious
> > > > > > > > > challenges, as replacing the metastore is generally just a
> > matter
> > > > > of
> > > > > > > > > changing the configuration, and making sure the realm is
> > properly
> > > > > > > > > bootstrapped.
> > > > > > > > >
> > > > > > > > > There will be, however, a few tricky situations in
> > polaris-core: a
> > > > > > few
> > > > > > > > > tests rely on the TreeMapMetaStore, mostly as a test
> > convenience;
> > > > > but
> > > > > > > > > there is no obvious replacement for it in that module. I am
> > however
> > > > > > > > > confident that we can find a solution for that, either based
> > on
> > > > > > mocks,
> > > > > > > > > or by bringing in a real metastore.
> > > > > > > > >
> > > > > > > > > If no objections are raised, I am going to prepare a PR for
> > this.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Alex
> > > > > > > > >
> > > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Alex,
> > > > > > > > > >
> > > > > > > > > > Thanks for starting this thread.
> > > > > > > > > >
> > > > > > > > > > I also find the TransactionalMetaStoreManagerImpl and
> > related
> > > > > call
> > > > > > paths
> > > > > > > > > > conuising in Apache Polaris code.
> > > > > > > > > >
> > > > > > > > > > I support promoting H2 to the default (in memory)
> > Persistence
> > > > > > backend for
> > > > > > > > > > getting started cases. This should also resolve the old H2
> > > > > > evolution
> > > > > > > > > thread
> > > > > > > > > > [1] by ensuring it is used regularly on the same code
> > paths as
> > > > > > > > > PostgreSQL.
> > > > > > > > > >
> > > > > > > > > > Deprecating TransactionalMetaStoreManagerImpl for removal
> > also
> > > > > > sounds
> > > > > > > > > > reasonable to me. Existing downstream users
> > > > > > > > > > of TransactionalMetaStoreManagerImpl will have time to
> > migrate,
> > > > > or
> > > > > > even
> > > > > > > > > > embed that code (per ASF license) into local builds,
> > during the
> > > > > > > > > deprecation
> > > > > > > > > > phase.
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre Dutra <
> > > > > > adutra@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I am writing to ask the community whether it is OK to
> > deprecate
> > > > > > > > > > > TreeMapMetaStore, as well as all the in-memory metastore
> > > > > manager,
> > > > > > > > > > > metastore manager factory, and persistence types that
> > rely
> > > > > > solely on
> > > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > > >
> > > > > > > > > > > As we know, these components are test-grade only, and not
> > > > > > suitable for
> > > > > > > > > > > production. They trigger a production readiness alert on
> > > > > Polaris
> > > > > > > > > > > startup. It's a considerable amount of code that is
> > virtually
> > > > > > dead in
> > > > > > > > > > > production.
> > > > > > > > > > >
> > > > > > > > > > > It's also confusing for developers. E.g. the
> > "transactional"
> > > > > > metastore
> > > > > > > > > > > is not transactional in the JDBC sense of the term, and
> > thus
> > > > > not
> > > > > > used
> > > > > > > > > > > by JDBC persistence. It also has its quirks: some return
> > > > > > statuses are
> > > > > > > > > > > only returned by that manager.
> > > > > > > > > > >
> > > > > > > > > > > However, these components are used in tests, and I agree
> > that
> > > > > > it's
> > > > > > > > > > > useful to have an in-memory version of the persistence
> > layer
> > > > > for
> > > > > > > > > > > tests. But we have today two alternatives that are imho
> > > > > superior
> > > > > > for
> > > > > > > > > > > tests in polaris-runtime-service:
> > > > > > > > > > >
> > > > > > > > > > > - JDBC persistence with H2 backend. There are already a
> > few
> > > > > tests
> > > > > > > > > > > using this setup.
> > > > > > > > > > > - NoSQL persistence with InMemory backend.
> > > > > > > > > > >
> > > > > > > > > > > Both alternatives test real production-grade persistence
> > code.
> > > > > > > > > > >
> > > > > > > > > > > And finally, TreeMapMetaStore is currently the default
> > runtime
> > > > > > > > > > > persistence in application.properties; and the Helm
> > chart also
> > > > > > > > > > > advertises it as the default. These are not sane
> > defaults,
> > > > > imho.
> > > > > > It's
> > > > > > > > > > > always tricky to provide a good default for datastores,
> > but
> > > > > > since JDBC
> > > > > > > > > > > persistence is included by default in the server image,
> > I think
> > > > > > that
> > > > > > > > > > > including the H2 driver by default could give us a saner
> > > > > default
> > > > > > while
> > > > > > > > > > > keeping the out-of-the-box experience intact (the
> > license is
> > > > > > Category
> > > > > > > > > > > A).
> > > > > > > > > > >
> > > > > > > > > > > Concretely:
> > > > > > > > > > >
> > > > > > > > > > > On the MetaStoreManagerFactory hierarchy, the following
> > > > > > > > > > > implementations are completely in-memory:
> > > > > > > > > > >
> > > > > > > > > > > - InMemoryPolarisMetaStoreManagerFactory: could be
> > removed
> > > > > > > > > > >
> > > > > > > > > > > - InMemoryAtomicOperationMetaStoreManagerFactory: could
> > be
> > > > > > removed
> > > > > > > > > > >
> > > > > > > > > > > - LocalPolarisMetaStoreManagerFactory: is the base class
> > of the
> > > > > > two
> > > > > > > > > > > above; imho it can be removed, but since it's an abstract
> > > > > class,
> > > > > > it
> > > > > > > > > > > may have been extended outside Polaris. But neither JDBC
> > nor
> > > > > > NoSQL use
> > > > > > > > > > > it.
> > > > > > > > > > >
> > > > > > > > > > > On the PolarisMetaStoreManager hierarchy:
> > > > > > > > > > >
> > > > > > > > > > > - TransactionalMetaStoreManagerImpl: is only used by
> > > > > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC and NoSQL do
> > not use
> > > > > > it.
> > > > > > > > > > > Could be removed.
> > > > > > > > > > >
> > > > > > > > > > > - TransactionWorkspaceMetaStoreManager however is a
> > different
> > > > > > beast,
> > > > > > > > > > > in spite of the similar name. It is in use today on the
> > commit
> > > > > > path,
> > > > > > > > > > > so should not be removed.
> > > > > > > > > > >
> > > > > > > > > > > On the BasePersistence hierarchy:
> > > > > > > > > > >
> > > > > > > > > > > - TreeMapTransactionalPersistenceImpl and its
> > TreeMapMetaStore
> > > > > > are
> > > > > > > > > > > only used by InMemoryPolarisMetaStoreManagerFactory, and
> > could
> > > > > be
> > > > > > > > > > > removed;
> > > > > > > > > > >
> > > > > > > > > > > - TransactionalPersistence and
> > > > > AbstractTransactionalPersistence:
> > > > > > these
> > > > > > > > > > > are supertypes of TreeMapTransactionalPersistenceImpl
> > and thus
> > > > > > only
> > > > > > > > > > > used for in-memory. They imo can be removed, but they
> > might
> > > > > have
> > > > > > been
> > > > > > > > > > > extended or implemented outside Polaris.
> > > > > > > > > > >
> > > > > > > > > > > I think it's important to keep Polaris code tidy by
> > removing
> > > > > > unused,
> > > > > > > > > > > unimplementable, or test-grade only components.
> > > > > > > > > > >
> > > > > > > > > > > What are your thoughts?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Alex
> > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > > >
> >
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi
Yeah fully agree and it was my point when I replied on the thread
initially.
Regards
JB
Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <fl...@gmail.com> a écrit :
> If our goal is to make the default experience closer to production
> while removing TreeMap, then JDBC with H2 seems like a more natural
> choice. It exercises the same persistence layer, schema management,
> and configuration that users will encounter in production, while still
> keeping setup lightweight.
>
> Yufei
>
> Yufei
>
>
> On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <ad...@apache.org> wrote:
> >
> > Hi Russell,
> >
> > The existing default metastore shouldn't be viewed merely as a
> > "relatively simple, self-contained implementation": it has unique
> > quirks, such as returning result types that no other metastore does.
> >
> > Choosing the status quo simply to avoid making changes is imho the
> > worst solution.
> >
> > Transitioning to the NoSQL metastore as the default remains a net
> > benefit, even if it is only used by 30% of the user base.
> >
> > However, I agree with keeping the test suite on JDBC + H2 by default
> > if that ensures a higher level of confidence in our tests ability to
> > reproduce real production setups.
> >
> > Thanks,
> > Alex
> >
> > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
> > <ru...@gmail.com> wrote:
> > >
> > > I agree TreeMap and its associated transactional stack should be
> deprecated.
> > > It’s test-only, confusing, and not used by JDBC production paths.
> > >
> > > I’m less convinced that NoSQL + InMemory is a clearly superior
> replacement
> > > for TreeMap in tests. At the storage layer it’s essentially the same
> class
> > > of
> > > thing: an in-memory, non-durable backend. The difference is that it
> > > exercises
> > > a completely different MetaStoreManager/persistence stack
> > > (NoSqlMetaStoreManager
> > > vs TransactionalMetaStoreManagerImpl), not the JDBC path most
> deployments
> > > use. It
> > > replaces a relatively simple, self-contained implementation with
> something
> > > that is
> > > part of a much more complicated persistence stack, while remaining
> > > non-durable
> > > and test-only at the storage layer.
> > >
> > > For the problems originally raised (test-grade defaults, dead
> production
> > > path, developer confusion),
> > > JDBC + H2 seems like an actual fix.
> > >
> > >
> > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Alex,
> > > >
> > > > I tend to agree that the DataSource discussion related to H2 became
> a bit
> > > > convoluted, and it's a lot simpler to use the NoSQL in-memory
> persistence
> > > > instead.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <ad...@apache.org>
> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Some updates on this:
> > > > >
> > > > > I recently started a separate discussion regarding dynamic
> datasource
> > > > > activation at runtime [1]. This was intended as a prerequisite for
> > > > > adopting H2 as the default JDBC driver, as suggested on this
> thread.
> > > > >
> > > > > However, that proposal may end up being rejected.
> > > > >
> > > > > Given this situation, we should reconsider our options. If the
> > > > > PostgreSQL + H2 approach is no longer a viable replacement to the
> > > > > TreeMapMetaStore, I propose an alternative: let's use the NoSQL
> > > > > metastore with an InMemory backend.
> > > > >
> > > > > My reasoning is as follows:
> > > > >
> > > > > 1. It remains a production-ready metastore, so strictly superior to
> > > > > TreeMapMetaStore.
> > > > >
> > > > > 2. The metastore state update logic is the same regardless of the
> > > > > backend, so we're still close to a real production setup (same as
> with
> > > > > an in-memory JDBC driver).
> > > > >
> > > > > 3. It doesn't require any external dependencies (no bundled
> driver).
> > > > >
> > > > > I'd also note that while the NoSQL metastore is already included
> > > > > today, it is not actionable because no backend is available. This
> goes
> > > > > against the onboarding UX that we're trying to achieve, as users
> > > > > interested in Polaris with the NoSQL metastore cannot test this
> setup
> > > > > with the official image.
> > > > >
> > > > > Therefore, I think NoSQL + InMemory would serve as a more
> appropriate
> > > > > default setup for the official Polaris image.
> > > > >
> > > > > From what I see, it would boil down to adding one line to
> > > > > polaris-server or polaris-service:
> > > > >
> > > > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > > > >
> > > > > What are your thoughts on this?
> > > > >
> > > > > Thanks,
> > > > > Alex
> > > > >
> > > > > [1]:
> https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > > > >
> > > > >
> > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra <
> adutra@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I wanted to give an update on this topic: I have started working
> on
> > > > > > this, but I would like to have [4812] merged first, since that's
> a
> > > > > > prerequisite.
> > > > > >
> > > > > > Thanks,
> > > > > > Alex
> > > > > >
> > > > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > > > >
> > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> > > > > > <ru...@gmail.com> wrote:
> > > > > > >
> > > > > > > I believe the original idea of TreeMapMetastore was that it
> looked
> > > > very
> > > > > > > similar to FoundationDB from an API perspective, so it served
> well
> > > > as
> > > > > a
> > > > > > > test system for the original backend being developed at SF.
> > > > > > >
> > > > > > > I agree that H2 + JDBC makes sense to me now for the project.
> > > > > > >
> > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra <
> adutra@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Thank you for your feedback.
> > > > > > > >
> > > > > > > > There seems to be a general agreement on the idea of
> deprecating
> > > > > > > > TreeMapMetaStore, coupled with the JDBC + H2 solution for
> tests.
> > > > > > > >
> > > > > > > > In most modules, the tests migration won't pose any serious
> > > > > > > > challenges, as replacing the metastore is generally just a
> matter
> > > > of
> > > > > > > > changing the configuration, and making sure the realm is
> properly
> > > > > > > > bootstrapped.
> > > > > > > >
> > > > > > > > There will be, however, a few tricky situations in
> polaris-core: a
> > > > > few
> > > > > > > > tests rely on the TreeMapMetaStore, mostly as a test
> convenience;
> > > > but
> > > > > > > > there is no obvious replacement for it in that module. I am
> however
> > > > > > > > confident that we can find a solution for that, either based
> on
> > > > > mocks,
> > > > > > > > or by bringing in a real metastore.
> > > > > > > >
> > > > > > > > If no objections are raised, I am going to prepare a PR for
> this.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Alex
> > > > > > > >
> > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Alex,
> > > > > > > > >
> > > > > > > > > Thanks for starting this thread.
> > > > > > > > >
> > > > > > > > > I also find the TransactionalMetaStoreManagerImpl and
> related
> > > > call
> > > > > paths
> > > > > > > > > conuising in Apache Polaris code.
> > > > > > > > >
> > > > > > > > > I support promoting H2 to the default (in memory)
> Persistence
> > > > > backend for
> > > > > > > > > getting started cases. This should also resolve the old H2
> > > > > evolution
> > > > > > > > thread
> > > > > > > > > [1] by ensuring it is used regularly on the same code
> paths as
> > > > > > > > PostgreSQL.
> > > > > > > > >
> > > > > > > > > Deprecating TransactionalMetaStoreManagerImpl for removal
> also
> > > > > sounds
> > > > > > > > > reasonable to me. Existing downstream users
> > > > > > > > > of TransactionalMetaStoreManagerImpl will have time to
> migrate,
> > > > or
> > > > > even
> > > > > > > > > embed that code (per ASF license) into local builds,
> during the
> > > > > > > > deprecation
> > > > > > > > > phase.
> > > > > > > > >
> > > > > > > > > [1]
> > > > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre Dutra <
> > > > > adutra@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I am writing to ask the community whether it is OK to
> deprecate
> > > > > > > > > > TreeMapMetaStore, as well as all the in-memory metastore
> > > > manager,
> > > > > > > > > > metastore manager factory, and persistence types that
> rely
> > > > > solely on
> > > > > > > > > > TreeMapMetaStore.
> > > > > > > > > >
> > > > > > > > > > As we know, these components are test-grade only, and not
> > > > > suitable for
> > > > > > > > > > production. They trigger a production readiness alert on
> > > > Polaris
> > > > > > > > > > startup. It's a considerable amount of code that is
> virtually
> > > > > dead in
> > > > > > > > > > production.
> > > > > > > > > >
> > > > > > > > > > It's also confusing for developers. E.g. the
> "transactional"
> > > > > metastore
> > > > > > > > > > is not transactional in the JDBC sense of the term, and
> thus
> > > > not
> > > > > used
> > > > > > > > > > by JDBC persistence. It also has its quirks: some return
> > > > > statuses are
> > > > > > > > > > only returned by that manager.
> > > > > > > > > >
> > > > > > > > > > However, these components are used in tests, and I agree
> that
> > > > > it's
> > > > > > > > > > useful to have an in-memory version of the persistence
> layer
> > > > for
> > > > > > > > > > tests. But we have today two alternatives that are imho
> > > > superior
> > > > > for
> > > > > > > > > > tests in polaris-runtime-service:
> > > > > > > > > >
> > > > > > > > > > - JDBC persistence with H2 backend. There are already a
> few
> > > > tests
> > > > > > > > > > using this setup.
> > > > > > > > > > - NoSQL persistence with InMemory backend.
> > > > > > > > > >
> > > > > > > > > > Both alternatives test real production-grade persistence
> code.
> > > > > > > > > >
> > > > > > > > > > And finally, TreeMapMetaStore is currently the default
> runtime
> > > > > > > > > > persistence in application.properties; and the Helm
> chart also
> > > > > > > > > > advertises it as the default. These are not sane
> defaults,
> > > > imho.
> > > > > It's
> > > > > > > > > > always tricky to provide a good default for datastores,
> but
> > > > > since JDBC
> > > > > > > > > > persistence is included by default in the server image,
> I think
> > > > > that
> > > > > > > > > > including the H2 driver by default could give us a saner
> > > > default
> > > > > while
> > > > > > > > > > keeping the out-of-the-box experience intact (the
> license is
> > > > > Category
> > > > > > > > > > A).
> > > > > > > > > >
> > > > > > > > > > Concretely:
> > > > > > > > > >
> > > > > > > > > > On the MetaStoreManagerFactory hierarchy, the following
> > > > > > > > > > implementations are completely in-memory:
> > > > > > > > > >
> > > > > > > > > > - InMemoryPolarisMetaStoreManagerFactory: could be
> removed
> > > > > > > > > >
> > > > > > > > > > - InMemoryAtomicOperationMetaStoreManagerFactory: could
> be
> > > > > removed
> > > > > > > > > >
> > > > > > > > > > - LocalPolarisMetaStoreManagerFactory: is the base class
> of the
> > > > > two
> > > > > > > > > > above; imho it can be removed, but since it's an abstract
> > > > class,
> > > > > it
> > > > > > > > > > may have been extended outside Polaris. But neither JDBC
> nor
> > > > > NoSQL use
> > > > > > > > > > it.
> > > > > > > > > >
> > > > > > > > > > On the PolarisMetaStoreManager hierarchy:
> > > > > > > > > >
> > > > > > > > > > - TransactionalMetaStoreManagerImpl: is only used by
> > > > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC and NoSQL do
> not use
> > > > > it.
> > > > > > > > > > Could be removed.
> > > > > > > > > >
> > > > > > > > > > - TransactionWorkspaceMetaStoreManager however is a
> different
> > > > > beast,
> > > > > > > > > > in spite of the similar name. It is in use today on the
> commit
> > > > > path,
> > > > > > > > > > so should not be removed.
> > > > > > > > > >
> > > > > > > > > > On the BasePersistence hierarchy:
> > > > > > > > > >
> > > > > > > > > > - TreeMapTransactionalPersistenceImpl and its
> TreeMapMetaStore
> > > > > are
> > > > > > > > > > only used by InMemoryPolarisMetaStoreManagerFactory, and
> could
> > > > be
> > > > > > > > > > removed;
> > > > > > > > > >
> > > > > > > > > > - TransactionalPersistence and
> > > > AbstractTransactionalPersistence:
> > > > > these
> > > > > > > > > > are supertypes of TreeMapTransactionalPersistenceImpl
> and thus
> > > > > only
> > > > > > > > > > used for in-memory. They imo can be removed, but they
> might
> > > > have
> > > > > been
> > > > > > > > > > extended or implemented outside Polaris.
> > > > > > > > > >
> > > > > > > > > > I think it's important to keep Polaris code tidy by
> removing
> > > > > unused,
> > > > > > > > > > unimplementable, or test-grade only components.
> > > > > > > > > >
> > > > > > > > > > What are your thoughts?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Alex
> > > > > > > > > >
> > > > > > > >
> > > > >
> > > >
>
Re: [DISCUSS] Deprecate TreeMapMetaStore and friends for removal
Posted by Yufei Gu <fl...@gmail.com>.
If our goal is to make the default experience closer to production
while removing TreeMap, then JDBC with H2 seems like a more natural
choice. It exercises the same persistence layer, schema management,
and configuration that users will encounter in production, while still
keeping setup lightweight.
Yufei
Yufei
On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <ad...@apache.org> wrote:
>
> Hi Russell,
>
> The existing default metastore shouldn't be viewed merely as a
> "relatively simple, self-contained implementation": it has unique
> quirks, such as returning result types that no other metastore does.
>
> Choosing the status quo simply to avoid making changes is imho the
> worst solution.
>
> Transitioning to the NoSQL metastore as the default remains a net
> benefit, even if it is only used by 30% of the user base.
>
> However, I agree with keeping the test suite on JDBC + H2 by default
> if that ensures a higher level of confidence in our tests ability to
> reproduce real production setups.
>
> Thanks,
> Alex
>
> On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer
> <ru...@gmail.com> wrote:
> >
> > I agree TreeMap and its associated transactional stack should be deprecated.
> > It’s test-only, confusing, and not used by JDBC production paths.
> >
> > I’m less convinced that NoSQL + InMemory is a clearly superior replacement
> > for TreeMap in tests. At the storage layer it’s essentially the same class
> > of
> > thing: an in-memory, non-durable backend. The difference is that it
> > exercises
> > a completely different MetaStoreManager/persistence stack
> > (NoSqlMetaStoreManager
> > vs TransactionalMetaStoreManagerImpl), not the JDBC path most deployments
> > use. It
> > replaces a relatively simple, self-contained implementation with something
> > that is
> > part of a much more complicated persistence stack, while remaining
> > non-durable
> > and test-only at the storage layer.
> >
> > For the problems originally raised (test-grade defaults, dead production
> > path, developer confusion),
> > JDBC + H2 seems like an actual fix.
> >
> >
> > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Alex,
> > >
> > > I tend to agree that the DataSource discussion related to H2 became a bit
> > > convoluted, and it's a lot simpler to use the NoSQL in-memory persistence
> > > instead.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <ad...@apache.org> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Some updates on this:
> > > >
> > > > I recently started a separate discussion regarding dynamic datasource
> > > > activation at runtime [1]. This was intended as a prerequisite for
> > > > adopting H2 as the default JDBC driver, as suggested on this thread.
> > > >
> > > > However, that proposal may end up being rejected.
> > > >
> > > > Given this situation, we should reconsider our options. If the
> > > > PostgreSQL + H2 approach is no longer a viable replacement to the
> > > > TreeMapMetaStore, I propose an alternative: let's use the NoSQL
> > > > metastore with an InMemory backend.
> > > >
> > > > My reasoning is as follows:
> > > >
> > > > 1. It remains a production-ready metastore, so strictly superior to
> > > > TreeMapMetaStore.
> > > >
> > > > 2. The metastore state update logic is the same regardless of the
> > > > backend, so we're still close to a real production setup (same as with
> > > > an in-memory JDBC driver).
> > > >
> > > > 3. It doesn't require any external dependencies (no bundled driver).
> > > >
> > > > I'd also note that while the NoSQL metastore is already included
> > > > today, it is not actionable because no backend is available. This goes
> > > > against the onboarding UX that we're trying to achieve, as users
> > > > interested in Polaris with the NoSQL metastore cannot test this setup
> > > > with the official image.
> > > >
> > > > Therefore, I think NoSQL + InMemory would serve as a more appropriate
> > > > default setup for the official Polaris image.
> > > >
> > > > From what I see, it would boil down to adding one line to
> > > > polaris-server or polaris-service:
> > > >
> > > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
> > > >
> > > > What are your thoughts on this?
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > > > [1]: https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
> > > >
> > > >
> > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra <ad...@apache.org>
> > > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I wanted to give an update on this topic: I have started working on
> > > > > this, but I would like to have [4812] merged first, since that's a
> > > > > prerequisite.
> > > > >
> > > > > Thanks,
> > > > > Alex
> > > > >
> > > > > [4812]: https://github.com/apache/polaris/pull/4812
> > > > >
> > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> > > > > <ru...@gmail.com> wrote:
> > > > > >
> > > > > > I believe the original idea of TreeMapMetastore was that it looked
> > > very
> > > > > > similar to FoundationDB from an API perspective, so it served well
> > > as
> > > > a
> > > > > > test system for the original backend being developed at SF.
> > > > > >
> > > > > > I agree that H2 + JDBC makes sense to me now for the project.
> > > > > >
> > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra <ad...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Thank you for your feedback.
> > > > > > >
> > > > > > > There seems to be a general agreement on the idea of deprecating
> > > > > > > TreeMapMetaStore, coupled with the JDBC + H2 solution for tests.
> > > > > > >
> > > > > > > In most modules, the tests migration won't pose any serious
> > > > > > > challenges, as replacing the metastore is generally just a matter
> > > of
> > > > > > > changing the configuration, and making sure the realm is properly
> > > > > > > bootstrapped.
> > > > > > >
> > > > > > > There will be, however, a few tricky situations in polaris-core: a
> > > > few
> > > > > > > tests rely on the TreeMapMetaStore, mostly as a test convenience;
> > > but
> > > > > > > there is no obvious replacement for it in that module. I am however
> > > > > > > confident that we can find a solution for that, either based on
> > > > mocks,
> > > > > > > or by bringing in a real metastore.
> > > > > > >
> > > > > > > If no objections are raised, I am going to prepare a PR for this.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Alex
> > > > > > >
> > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Alex,
> > > > > > > >
> > > > > > > > Thanks for starting this thread.
> > > > > > > >
> > > > > > > > I also find the TransactionalMetaStoreManagerImpl and related
> > > call
> > > > paths
> > > > > > > > conuising in Apache Polaris code.
> > > > > > > >
> > > > > > > > I support promoting H2 to the default (in memory) Persistence
> > > > backend for
> > > > > > > > getting started cases. This should also resolve the old H2
> > > > evolution
> > > > > > > thread
> > > > > > > > [1] by ensuring it is used regularly on the same code paths as
> > > > > > > PostgreSQL.
> > > > > > > >
> > > > > > > > Deprecating TransactionalMetaStoreManagerImpl for removal also
> > > > sounds
> > > > > > > > reasonable to me. Existing downstream users
> > > > > > > > of TransactionalMetaStoreManagerImpl will have time to migrate,
> > > or
> > > > even
> > > > > > > > embed that code (per ASF license) into local builds, during the
> > > > > > > deprecation
> > > > > > > > phase.
> > > > > > > >
> > > > > > > > [1]
> > > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre Dutra <
> > > > adutra@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I am writing to ask the community whether it is OK to deprecate
> > > > > > > > > TreeMapMetaStore, as well as all the in-memory metastore
> > > manager,
> > > > > > > > > metastore manager factory, and persistence types that rely
> > > > solely on
> > > > > > > > > TreeMapMetaStore.
> > > > > > > > >
> > > > > > > > > As we know, these components are test-grade only, and not
> > > > suitable for
> > > > > > > > > production. They trigger a production readiness alert on
> > > Polaris
> > > > > > > > > startup. It's a considerable amount of code that is virtually
> > > > dead in
> > > > > > > > > production.
> > > > > > > > >
> > > > > > > > > It's also confusing for developers. E.g. the "transactional"
> > > > metastore
> > > > > > > > > is not transactional in the JDBC sense of the term, and thus
> > > not
> > > > used
> > > > > > > > > by JDBC persistence. It also has its quirks: some return
> > > > statuses are
> > > > > > > > > only returned by that manager.
> > > > > > > > >
> > > > > > > > > However, these components are used in tests, and I agree that
> > > > it's
> > > > > > > > > useful to have an in-memory version of the persistence layer
> > > for
> > > > > > > > > tests. But we have today two alternatives that are imho
> > > superior
> > > > for
> > > > > > > > > tests in polaris-runtime-service:
> > > > > > > > >
> > > > > > > > > - JDBC persistence with H2 backend. There are already a few
> > > tests
> > > > > > > > > using this setup.
> > > > > > > > > - NoSQL persistence with InMemory backend.
> > > > > > > > >
> > > > > > > > > Both alternatives test real production-grade persistence code.
> > > > > > > > >
> > > > > > > > > And finally, TreeMapMetaStore is currently the default runtime
> > > > > > > > > persistence in application.properties; and the Helm chart also
> > > > > > > > > advertises it as the default. These are not sane defaults,
> > > imho.
> > > > It's
> > > > > > > > > always tricky to provide a good default for datastores, but
> > > > since JDBC
> > > > > > > > > persistence is included by default in the server image, I think
> > > > that
> > > > > > > > > including the H2 driver by default could give us a saner
> > > default
> > > > while
> > > > > > > > > keeping the out-of-the-box experience intact (the license is
> > > > Category
> > > > > > > > > A).
> > > > > > > > >
> > > > > > > > > Concretely:
> > > > > > > > >
> > > > > > > > > On the MetaStoreManagerFactory hierarchy, the following
> > > > > > > > > implementations are completely in-memory:
> > > > > > > > >
> > > > > > > > > - InMemoryPolarisMetaStoreManagerFactory: could be removed
> > > > > > > > >
> > > > > > > > > - InMemoryAtomicOperationMetaStoreManagerFactory: could be
> > > > removed
> > > > > > > > >
> > > > > > > > > - LocalPolarisMetaStoreManagerFactory: is the base class of the
> > > > two
> > > > > > > > > above; imho it can be removed, but since it's an abstract
> > > class,
> > > > it
> > > > > > > > > may have been extended outside Polaris. But neither JDBC nor
> > > > NoSQL use
> > > > > > > > > it.
> > > > > > > > >
> > > > > > > > > On the PolarisMetaStoreManager hierarchy:
> > > > > > > > >
> > > > > > > > > - TransactionalMetaStoreManagerImpl: is only used by
> > > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC and NoSQL do not use
> > > > it.
> > > > > > > > > Could be removed.
> > > > > > > > >
> > > > > > > > > - TransactionWorkspaceMetaStoreManager however is a different
> > > > beast,
> > > > > > > > > in spite of the similar name. It is in use today on the commit
> > > > path,
> > > > > > > > > so should not be removed.
> > > > > > > > >
> > > > > > > > > On the BasePersistence hierarchy:
> > > > > > > > >
> > > > > > > > > - TreeMapTransactionalPersistenceImpl and its TreeMapMetaStore
> > > > are
> > > > > > > > > only used by InMemoryPolarisMetaStoreManagerFactory, and could
> > > be
> > > > > > > > > removed;
> > > > > > > > >
> > > > > > > > > - TransactionalPersistence and
> > > AbstractTransactionalPersistence:
> > > > these
> > > > > > > > > are supertypes of TreeMapTransactionalPersistenceImpl and thus
> > > > only
> > > > > > > > > used for in-memory. They imo can be removed, but they might
> > > have
> > > > been
> > > > > > > > > extended or implemented outside Polaris.
> > > > > > > > >
> > > > > > > > > I think it's important to keep Polaris code tidy by removing
> > > > unused,
> > > > > > > > > unimplementable, or test-grade only components.
> > > > > > > > >
> > > > > > > > > What are your thoughts?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Alex
> > > > > > > > >
> > > > > > >
> > > >
> > >